search

dynverse / dyno

Inferring, interpreting and visualising trajectories using a streamlined set of packages 🦕
https://dynverse.github.io/dyno
Other
163 stars 32 forks source link

Trouble with dyno on HPC #71

Closed rhodesch closed 1 year ago

rhodesch commented 4 years ago

Hi,

Great tool!

I'm having trouble running dyno/singularity on an hpc cluster. There seem to be a number of previous issues related to using dyno with singularity on an hpc, but I haven't had much luck with any of the suggestions. I successfully ran similar code locally with docker. Any input would be greatly appreciated

require(devtools) devtools::install_github("dynverse/babelwhale")

dynwrap::test_singularity_installation(detailed = TRUE)

dynwrap::test_singularity_installation(detailed = TRUE) ✔ Singularity is installed ✔ Singularity is at correct version (>=3.0): 3.5.2 is installed ✔ Singularity can pull and run a container from Dockerhub ✔ Singularity can mount temporary volumes ✔ Singularity test successful ------------------------------------------------------------ [1] TRUE

library(dyno) data("fibroblast_reprogramming_treutlein")

babelwhale::set_default_config(babelwhale::create_singularity_config())

dataset <- wrap_expression( counts = fibroblast_reprogramming_treutlein$counts, expression = fibroblast_reprogramming_treutlein$expression )

model <- infer_trajectory(dataset, ti_slingshot(), verbose = TRUE)

model <- infer_trajectory(dataset, ti_slingshot(), verbose = TRUE) Running singularity exec 'docker://dynverse/ti_slingshot:v1.0.3' echo hi Running /usr/local/current/singularity/3.5.2/bin/singularity exec --containall -B \ '/tmp/RtmpcTpriZ/file23431f1fd28e/:/copy_mount,/tmp/RtmpcTpriZ/file23435440347d/tmp:/tmp2' \ 'docker://dynverse/ti_slingshot:v1.0.3' cp /code/definition.yml /copy_mount/ FATAL: Unable to handle docker://dynverse/ti_slingshot:v1.0.3 uri: failed to get SHA of docker://dynverse/ti_slingshot:v1.0.3: pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:52161->[::1]:53: read: connection refused Error: FATAL: Unable to handle docker://dynverse/ti_slingshot:v1.0.3 uri: failed to get SHA of docker://dynverse/ti_slingshot:v1.0.3: pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:52161->[::1]:53: read: connection refused

sessionInfo() R version 3.6.1 (2019-07-05) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core)

Matrix products: default BLAS/LAPACK: /usr/local/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_rt.so

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] dyno_0.1.1 dynwrap_1.1.4 dynplot_1.0.2.9000 dynmethods_1.0.5 dynguidelines_1.0.0
[6] dynfeature_1.0.0.9000 devtools_2.2.1 usethis_1.5.1

loaded via a namespace (and not attached): [1] colorspace_1.4-1 ellipsis_0.3.0 rprojroot_1.3-2 dynparam_1.0.0 htmlTable_1.13.3
[6] base64enc_0.1-3 fs_1.3.1 rje_1.10.13 rstudioapi_0.10 farver_2.0.1
[11] remotes_2.1.0 graphlayouts_0.5.0 dynutils_1.0.4 ggrepel_0.8.1 fansi_0.4.1
[16] ranger_0.11.2 codetools_0.2-16 splines_3.6.1 knitr_1.26 polyclip_1.10-0
[21] pkgload_1.0.2 zeallot_0.1.0 Formula_1.2-3 cluster_2.1.0 png_0.1-7
[26] ggforce_0.3.1 shiny_1.4.0 readr_1.3.1 compiler_3.6.1 backports_1.1.5
[31] assertthat_0.2.1 Matrix_1.2-18 fastmap_1.0.1 lazyeval_0.2.2 cli_2.0.1
[36] later_1.0.0 tweenr_1.0.1 acepack_1.4.1 htmltools_0.4.0 prettyunits_1.1.0
[41] tools_3.6.1 igraph_1.2.4.2 gtable_0.3.0 glue_1.3.1 reshape2_1.4.3
[46] dplyr_0.8.3 Rcpp_1.0.3 GA_3.2 vctrs_0.2.1 iterators_1.0.12
[51] ggraph_2.0.0 xfun_0.12 stringr_1.4.0 ps_1.3.0 testthat_2.3.1
[56] mime_0.8 lifecycle_0.1.0 irlba_2.3.3 MASS_7.3-51.4 scales_1.1.0
[61] tidygraph_1.1.2 babelwhale_1.0.1 hms_0.5.3 promises_1.1.0 RColorBrewer_1.1-2
[66] yaml_2.2.0 curl_4.3 memoise_1.1.0 gridExtra_2.3 ggplot2_3.2.1
[71] dyndimred_1.0.2.9000 rpart_4.1-15 latticeExtra_0.6-29 stringi_1.4.5 desc_1.2.0
[76] foreach_1.4.7 checkmate_1.9.4 pkgbuild_1.0.6 rlang_0.4.2 pkgconfig_2.0.3
[81] lattice_0.20-38 purrr_0.3.3 patchwork_1.0.0 htmlwidgets_1.5.1 cowplot_1.0.0
[86] processx_3.4.1 tidyselect_0.2.5 plyr_1.8.5 magrittr_1.5 R6_2.4.1
[91] Hmisc_4.3-0 pillar_1.4.3 foreign_0.8-73 carrier_0.1.0 withr_2.1.2
[96] proxyC_0.1.5 survival_3.1-8 nnet_7.3-12 tibble_2.1.3 crayon_1.3.4
[101] lmds_0.1.0 viridis_0.5.1 jpeg_0.1-8.1 grid_3.6.1 data.table_1.12.8
[106] callr_3.4.0 digest_0.6.23 xtable_1.8-4 tidyr_1.0.0 httpuv_1.5.2
[111] RcppParallel_4.4.4 munsell_0.5.0 viridisLite_0.3.0 sessioninfo_1.1.1

zouter commented 4 years ago

Hmmmm, seems like most steps went fine, except the connection with dockerhub:

Error: FATAL: Unable to handle docker://dynverse/ti_slingshot:v1.0.3 uri: failed to get SHA of docker://dynverse/ti_slingshot:v1.0.3: pinging docker registry returned: Get registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:52161->[::1]:53: read: connection refused

Could you try to do something like singularity pull docker://godlovedc/lolcow and see what this says?

rhodesch commented 4 years ago

Sure. singularity pull docker://godlovedc/lolcow -Adds a lolcow_latest.sif file to my $HOME directory.

$ singularity pull docker://godlovedc/lolcow INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob 9fb6c798fa41 done Copying blob 3b61febd4aef done Copying blob 9d99b9777eb0 done Copying blob d010c8cf75d7 done Copying blob 7fac07fb303e done Copying blob 8e860504ff1e done Copying config 73d5b1025f done Writing manifest to image destination Storing signatures 2020/01/14 17:35:06 info unpack layer: sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118 2020/01/14 17:35:06 warn rootless{dev/agpgart} creating empty file in place of device 10:175 2020/01/14 17:35:06 warn rootless{dev/audio} creating empty file in place of device 14:4 2020/01/14 17:35:06 warn rootless{dev/audio1} creating empty file in place of device 14:20 2020/01/14 17:35:06 warn rootless{dev/audio2} creating empty file in place of device 14:36 2020/01/14 17:35:06 warn rootless{dev/audio3} creating empty file in place of device 14:52 2020/01/14 17:35:06 warn rootless{dev/audioctl} creating empty file in place of device 14:7 2020/01/14 17:35:06 warn rootless{dev/console} creating empty file in place of device 5:1 2020/01/14 17:35:06 warn rootless{dev/dsp} creating empty file in place of device 14:3 2020/01/14 17:35:06 warn rootless{dev/dsp1} creating empty file in place of device 14:19 2020/01/14 17:35:06 warn rootless{dev/dsp2} creating empty file in place of device 14:35 2020/01/14 17:35:06 warn rootless{dev/dsp3} creating empty file in place of device 14:51 2020/01/14 17:35:06 warn rootless{dev/full} creating empty file in place of device 1:7 2020/01/14 17:35:06 warn rootless{dev/kmem} creating empty file in place of device 1:2 2020/01/14 17:35:06 warn rootless{dev/loop0} creating empty file in place of device 7:0 2020/01/14 17:35:06 warn rootless{dev/loop1} creating empty file in place of device 7:1 2020/01/14 17:35:06 warn rootless{dev/loop2} creating empty file in place of device 7:2 2020/01/14 17:35:06 warn rootless{dev/loop3} creating empty file in place of device 7:3 2020/01/14 17:35:06 warn rootless{dev/loop4} creating empty file in place of device 7:4 2020/01/14 17:35:06 warn rootless{dev/loop5} creating empty file in place of device 7:5 2020/01/14 17:35:06 warn rootless{dev/loop6} creating empty file in place of device 7:6 2020/01/14 17:35:06 warn rootless{dev/loop7} creating empty file in place of device 7:7 2020/01/14 17:35:06 warn rootless{dev/mem} creating empty file in place of device 1:1 2020/01/14 17:35:06 warn rootless{dev/midi0} creating empty file in place of device 35:0 2020/01/14 17:35:06 warn rootless{dev/midi00} creating empty file in place of device 14:2 2020/01/14 17:35:06 warn rootless{dev/midi01} creating empty file in place of device 14:18 2020/01/14 17:35:06 warn rootless{dev/midi02} creating empty file in place of device 14:34 2020/01/14 17:35:06 warn rootless{dev/midi03} creating empty file in place of device 14:50 2020/01/14 17:35:06 warn rootless{dev/midi1} creating empty file in place of device 35:1 2020/01/14 17:35:06 warn rootless{dev/midi2} creating empty file in place of device 35:2 2020/01/14 17:35:06 warn rootless{dev/midi3} creating empty file in place of device 35:3 2020/01/14 17:35:06 warn rootless{dev/mixer} creating empty file in place of device 14:0 2020/01/14 17:35:06 warn rootless{dev/mixer1} creating empty file in place of device 14:16 2020/01/14 17:35:06 warn rootless{dev/mixer2} creating empty file in place of device 14:32 2020/01/14 17:35:06 warn rootless{dev/mixer3} creating empty file in place of device 14:48 2020/01/14 17:35:06 warn rootless{dev/mpu401data} creating empty file in place of device 31:0 2020/01/14 17:35:06 warn rootless{dev/mpu401stat} creating empty file in place of device 31:1 2020/01/14 17:35:06 warn rootless{dev/null} creating empty file in place of device 1:3 2020/01/14 17:35:06 warn rootless{dev/port} creating empty file in place of device 1:4 2020/01/14 17:35:06 warn rootless{dev/ram0} creating empty file in place of device 1:0 2020/01/14 17:35:06 warn rootless{dev/ram1} creating empty file in place of device 1:1 2020/01/14 17:35:06 warn rootless{dev/ram10} creating empty file in place of device 1:10 2020/01/14 17:35:06 warn rootless{dev/ram11} creating empty file in place of device 1:11 2020/01/14 17:35:06 warn rootless{dev/ram12} creating empty file in place of device 1:12 2020/01/14 17:35:06 warn rootless{dev/ram13} creating empty file in place of device 1:13 2020/01/14 17:35:06 warn rootless{dev/ram14} creating empty file in place of device 1:14 2020/01/14 17:35:06 warn rootless{dev/ram15} creating empty file in place of device 1:15 2020/01/14 17:35:06 warn rootless{dev/ram16} creating empty file in place of device 1:16 2020/01/14 17:35:06 warn rootless{dev/ram2} creating empty file in place of device 1:2 2020/01/14 17:35:06 warn rootless{dev/ram3} creating empty file in place of device 1:3 2020/01/14 17:35:06 warn rootless{dev/ram4} creating empty file in place of device 1:4 2020/01/14 17:35:06 warn rootless{dev/ram5} creating empty file in place of device 1:5 2020/01/14 17:35:06 warn rootless{dev/ram6} creating empty file in place of device 1:6 2020/01/14 17:35:06 warn rootless{dev/ram7} creating empty file in place of device 1:7 2020/01/14 17:35:06 warn rootless{dev/ram8} creating empty file in place of device 1:8 2020/01/14 17:35:06 warn rootless{dev/ram9} creating empty file in place of device 1:9 2020/01/14 17:35:06 warn rootless{dev/random} creating empty file in place of device 1:8 2020/01/14 17:35:06 warn rootless{dev/rmidi0} creating empty file in place of device 35:64 2020/01/14 17:35:06 warn rootless{dev/rmidi1} creating empty file in place of device 35:65 2020/01/14 17:35:06 warn rootless{dev/rmidi2} creating empty file in place of device 35:66 2020/01/14 17:35:06 warn rootless{dev/rmidi3} creating empty file in place of device 35:67 2020/01/14 17:35:06 warn rootless{dev/sequencer} creating empty file in place of device 14:1 2020/01/14 17:35:06 warn rootless{dev/smpte0} creating empty file in place of device 35:128 2020/01/14 17:35:06 warn rootless{dev/smpte1} creating empty file in place of device 35:129 2020/01/14 17:35:06 warn rootless{dev/smpte2} creating empty file in place of device 35:130 2020/01/14 17:35:06 warn rootless{dev/smpte3} creating empty file in place of device 35:131 2020/01/14 17:35:06 warn rootless{dev/sndstat} creating empty file in place of device 14:6 2020/01/14 17:35:06 warn rootless{dev/tty} creating empty file in place of device 5:0 2020/01/14 17:35:06 warn rootless{dev/tty0} creating empty file in place of device 4:0 2020/01/14 17:35:06 warn rootless{dev/tty1} creating empty file in place of device 4:1 2020/01/14 17:35:06 warn rootless{dev/tty2} creating empty file in place of device 4:2 2020/01/14 17:35:06 warn rootless{dev/tty3} creating empty file in place of device 4:3 2020/01/14 17:35:06 warn rootless{dev/tty4} creating empty file in place of device 4:4 2020/01/14 17:35:06 warn rootless{dev/tty5} creating empty file in place of device 4:5 2020/01/14 17:35:06 warn rootless{dev/tty6} creating empty file in place of device 4:6 2020/01/14 17:35:06 warn rootless{dev/tty7} creating empty file in place of device 4:7 2020/01/14 17:35:06 warn rootless{dev/tty8} creating empty file in place of device 4:8 2020/01/14 17:35:06 warn rootless{dev/tty9} creating empty file in place of device 4:9 2020/01/14 17:35:06 warn rootless{dev/urandom} creating empty file in place of device 1:9 2020/01/14 17:35:06 warn rootless{dev/zero} creating empty file in place of device 1:5 2020/01/14 17:35:08 info unpack layer: sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a 2020/01/14 17:35:08 info unpack layer: sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2 2020/01/14 17:35:08 info unpack layer: sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e 2020/01/14 17:35:08 info unpack layer: sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9 2020/01/14 17:35:08 info unpack layer: sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945 INFO: Creating SIF file... INFO: Build complete: lolcow_latest.sif

singularity run lolcow_latest.sif -Shows the cow.

I was able to pull a few other docker images form docker hub, each resulting in a .sif file appearing at my home dir. However, I could not sucessfully pull/build anything from dynverse:

builds SIF

singularity pull docker://couchbase

builds SIF

singularity pull docker://dynverse/ti_slice

builds SIF

singularity pull docker://hello-world

builds SIF

singularity pull docker://postgres

Fails

singularity pull docker://dynverse/ti_slingshot $ singularity pull docker://dynverse/ti_slingshot INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob a4d8138d0f6b skipped: already exists Copying blob fb232e1cbf74 skipped: already exists Copying blob 53a696448921 skipped: already exists Copying blob 0a98bf8e346b skipped: already exists Copying blob 80070ed7cf88 skipped: already exists Copying blob 3dda0c19343b skipped: already exists Copying blob c8dedce96c17 skipped: already exists Copying blob 33199d4ff47f skipped: already exists Copying blob 7d44151ac2da skipped: already exists Copying blob 71c13904cc33 skipped: already exists Copying blob a03a650d0366 skipped: already exists Copying blob d0e24af2427d skipped: already exists Copying blob 509c796290f5 skipped: already exists Copying blob fc7b41c3606c skipped: already exists Copying blob a0885e753793 skipped: already exists Copying config 9d4bf56ab1 done Writing manifest to image destination Storing signatures 2020/01/14 17:01:51 info unpack layer: sha256:a4d8138d0f6b5a441aaa533faf5fe0c3996a6ca42643c46f4402c7e8bda53742 2020/01/14 17:01:52 warn rootless{usr/local/man} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" 2020/01/14 17:01:53 info unpack layer: sha256:fb232e1cbf74f0fbcc4777cbfa268e7edacec9b944284fecb9a49a0717727371 2020/01/14 17:01:58 warn rootless{usr/local/bin/install2.r} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" 2020/01/14 17:01:58 warn rootless{usr/local/bin/installGithub.r} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" 2020/01/14 17:01:58 warn rootless{usr/local/bin/r} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" 2020/01/14 17:02:02 info unpack layer: sha256:53a69644892140b394d496f0206f1e7caaffe83ac5c60245721b2d00189e0914 2020/01/14 17:02:11 warn rootless{usr/local/bin/pandoc} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" 2020/01/14 17:02:11 warn rootless{usr/local/bin/pandoc-citeproc} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers" 2020/01/14 17:02:12 info unpack layer: sha256:0a98bf8e346bef6e92b33f0f28c5a473b6248a9ceb21686c5139485961033a8e 2020/01/14 17:02:12 info unpack layer: sha256:80070ed7cf8857a7fa54e7506039f1dc8c03beb2ce337fb45c2c7a963d6c3336 2020/01/14 17:02:12 info unpack layer: sha256:3dda0c19343b691057ef5913c40aa79e2d65e2ffb9cfead972003f055c52854a 2020/01/14 17:02:12 info unpack layer: sha256:c8dedce96c17f919cd38e6d135303d3f2ca29bf0f73245671e1e3693bfa3169f 2020/01/14 17:02:12 info unpack layer: sha256:33199d4ff47fef1ac4cd988b1a9e63341436fdfa00e6de3578d31031555385be 2020/01/14 17:02:26 info unpack layer: sha256:7d44151ac2dae4a5c213f71c964418e5a18472aef650a4e3add71164e1d4ac82 2020/01/14 17:02:27 info unpack layer: sha256:71c13904cc333688f9fa770fd95965f827949ea5c746b4dbb03e546fec3ad006 2020/01/14 17:02:27 info unpack layer: sha256:a03a650d036654b42695ee1f175f5780d39ab41a25caebbd65a999aa946bc4e4 2020/01/14 17:02:29 info unpack layer: sha256:d0e24af2427d84970f842be99c9e7320bebd79c6c408e05fdc8f98f5774cac4f 2020/01/14 17:02:33 info unpack layer: sha256:509c796290f5400effd645bb08a804ec8ce8587687552a105d0a94b0f984208d 2020/01/14 17:02:39 info unpack layer: sha256:fc7b41c3606c7d0cf3e66f4c7211f7ef0613e53fea45b733287465499f0bd216 2020/01/14 17:02:39 info unpack layer: sha256:a0885e753793697a00322473263d6dea5c81d582b626c8cc3e6e34116de37a0b INFO: Creating SIF file... FATAL: While making image from oci registry: while building SIF from layers: while creating squashfs: create command failed: signal: killed:

I was able to pull the ti_slingshot:v1.0.3 image within docker on a macOS.

rhodesch commented 4 years ago

I can now pull and build dynverse images such as docker://dynverse/ti_paga and docker://dynverse/ti_slingshot, as well as docker://godlovedc/lolcow. Turns out I was not allocating enough memory to my interactive cluster job, which was killing the pull/build processes.

I am now getting a new error message (see below). As I have pulled the dynmethods, can I just include the path of the SIF file (or directory containing SIF) such as $HOME/ti_paga_latest.sif in the infer_trajectory() method?

I tried: library(dyno) dynwrap::test_singularity_installation(detailed = TRUE) dynwrap::set_default_config(dynwrap::create_singularity_config(cache_dir = ".singularity/")) data("fibroblast_reprogramming_treutlein") dataset <- wrap_expression( counts = fibroblast_reprogramming_treutlein$counts, expression = fibroblast_reprogramming_treutlein$expression ) model <- infer_trajectory(dataset, "slingshot")

output:Running singularity exec 'docker://dynverse/ti_slingshot:v1.0.3' echo hi Running /usr/local/current/singularity/3.5.2/bin/singularity exec --containall -B \ '/tmp/RtmpfSl7W8/file421426476fa6/:/copy_mount,/tmp/RtmpfSl7W8/file42146370f3e/tmp:/tmp2' \ 'docker://dynverse/ti_slingshot:v1.0.3' cp /code/definition.yml /copy_mount/ FATAL: Unable to handle docker://dynverse/ti_slingshot:v1.0.3 uri: failed to get SHA of docker://dynverse/ti_slingshot:v1.0.3: pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:42166->[::1]:53: read: connection refused Error: FATAL: Unable to handle docker://dynverse/ti_slingshot:v1.0.3 uri: failed to get SHA of docker://dynverse/ti_slingshot:v1.0.3: pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:42166->[::1]:53: read: connection refused