v_simple of MASH module not recognizing the container

[hsun3163]

while running:

nohup sos run pipeline/mashr.ipynb mash \
    --container containers/stephenslab.sif \
    --output_prefix Ast_Exc_Inh_Mic_OPC_Oli \
    --data output/Ast_Exc_Inh_Mic_OPC_Oli.rds \
    -s force -J 20 -q csg -c csg.yml --cwd MASH_6_celltypes

Following error occurs:

ERROR: [vhat_simple (/mnt/vast/hpc/csg/snuc_pseu]: [0]: Executing script in Singularity returns an error (exitcode=255, stderr=/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH_6_celltypes/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_simple.stderr).
The script has been saved to /home/hs3163/.sos/865d80eff1b4b415/singularity_run_45740.R. To reproduce the error please run:
singularity exec  /home/hs3163/.sos/singularity/library/containers/stephenslab.sif Rscript /home/hs3163/.sos/865d80eff1b4b415/singularity_run_45740.R
[mash]: Exits with 3 pending steps (mash_1, mash_2, prior)

FATAL:   could not open image /home/hs3163/.sos/singularity/library/containers/stephenslab.sif: failed to retrieve path for /home/hs3163/.sos/singularity/library/containers/stephenslab.sif: lstat /mnt/mfs/hgrcgrid/homes/hs3163/.sos/singularity/library/containers: no such file or directory

The problem happened because the vhat_simple step use the wrong path for the singularity image, exemplified by singularity exec /home/hs3163/.sos/singularity/library/containers/stephenslab.sif. However, this problem didn't occur in other steps.

[hsun3163]

After deleting the ~/.sos dir, the error persist

[hsun3163]

It is rather strange that, after specifying the --vhat to use "mle" method. vhat_simple was still called:

nohup sos run pipeline/mashr.ipynb mash     --container containers/stephenslab.sif     --output_prefix Ast_Exc_Inh_Mic_OPC_Oli     --data output/Ast_Exc_Inh_Mic_OPC_Oli.rds     -s force -J 20 -q csg -c csg.yml --cwd MASH_6_celltypes --vhat mle  &

INFO: Running vhat_mle: V estimate: "mle" method
INFO: Running vhat_simple: V estimate: "simple" method (using null z-scores)
INFO: tdc5f08c88e4719fe restart from status failed
INFO: tdc5f08c88e4719fe submitted to csg with job id 2681715

[hsun3163]

Dryrun result indicates that anything involved with the vhat are running the incorrect singularity image

INFO: Checking flash_nonneg: Perform FLASH analysis with non-negative factor constraint (time estimate: 20min)
INFO: Checking pca:
INFO: Checking flash: Perform FLASH analysis with non-negative factor constraint (time estimate: 20min)
HINT: singularity exec  containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/tmpiganu7u8/singularity_run_46249.R
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
dat = mashr::mash_set_data(dat$strong.b, Shat=dat$strong.s, alpha=1, zero_Bhat_Shat_reset = 1E3)
res = mashr::cov_flash(dat, factors="nonneg", remove_singleton=TRUE, output_model="/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash_nonneg.model.rds")
saveRDS(res, '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash_nonneg.rds')

INFO: flash_nonneg is completed.
INFO: flash_nonneg output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash_nonneg.rds
HINT: singularity exec  containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/tmp140gxvgo/singularity_run_46266.R
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
dat = mashr::mash_set_data(dat$strong.b, Shat=dat$strong.s, alpha=1, zero_Bhat_Shat_reset = 1E3)
res = mashr::cov_pca(dat, 2)
saveRDS(res, '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.pca.rds')

HINT: singularity exec  containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/tmpv00w4lfc/singularity_run_46267.R
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
dat = mashr::mash_set_data(dat$strong.b, Shat=dat$strong.s, alpha=1, zero_Bhat_Shat_reset = 1E3)
res = mashr::cov_flash(dat, factors="default", remove_singleton=TRUE, output_model="/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash.model.rds")
saveRDS(res, '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash.rds')

INFO: pca is completed.
INFO: flash is completed.
INFO: pca output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.pca.rds
INFO: flash output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash.rds
INFO: Checking vhat_mle: V estimate: "mle" method
INFO: Checking vhat_simple: V estimate: "simple" method (using null z-scores)
HINT: singularity exec  /home/hs3163/.sos/singularity/library/containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/tmpg_x990gq/singularity_run_46272.R
library(mashr)
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
vhat = estimate_null_correlation_simple(mash_set_data(dat$random.b, Shat=dat$random.s, alpha=1, zero_Bhat_Shat_reset = 1E3))
saveRDS(vhat, '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_simple.rds')

INFO: vhat_simple is completed.
INFO: vhat_simple output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_simple.rds
INFO: Checking canonical:
HINT: singularity exec  containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/tmpi28lnkua/singularity_run_46249.R
library("mashr")
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
dat = mashr::mash_set_data(dat$strong.b, Shat=dat$strong.s, alpha=1, zero_Bhat_Shat_reset = 1E3)
res = mashr::cov_canonical(dat)
saveRDS(res, '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.canonical.rds')

INFO: canonical is completed.
INFO: canonical output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.canonical.rds
INFO: Checking prior: Compute data-driven / canonical prior matrices (time estimate: 2h ~ 12h for ~30 49 by 49 matrix mixture)
HINT: singularity exec  /home/hs3163/.sos/singularity/library/containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/tmp7u52xsge/singularity_run_46266.R
library(mashr)
rds_files = c('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds','/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_simple.rds','/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash.rds','/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.flash_nonneg.rds','/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.pca.rds','/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.canonical.rds')
dat = readRDS(rds_files[1])
vhat = readRDS(rds_files[2])
mash_data = mash_set_data(dat$strong.b, Shat=dat$strong.s, V=vhat, alpha=1, zero_Bhat_Shat_reset = 1E3)
# setup prior
U = list(XtX = t(mash_data$Bhat) %*% mash_data$Bhat / nrow(mash_data$Bhat))
for (f in rds_files[3:length(rds_files)]) U = c(U, readRDS(f))
U.ed = cov_ed(mash_data, U, logfile='/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.prior')
# Canonical matrices
U.can = cov_canonical(mash_data)
saveRDS(c(U.ed, U.can), '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.prior.rds')

INFO: prior is completed.
INFO: prior output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.prior.rds
HINT: singularity exec  /home/hs3163/.sos/singularity/library/containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/tmpva21tybx/singularity_run_46274.R
library(mashr)
# choose random subset
set.seed(1)
random.subset = sample(1:nrow(dat$random.b), min(6000, nrow(dat$random.b)))
random.subset = mash_set_data(dat$random.b[random.subset,], dat$random.s[random.subset,], alpha=1, zero_Bhat_Shat_reset = 1E3)
# estimate V mle
vhat = mash_estimate_corr_em(random.subset, readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.prior.rds'), max_iter = 6)
saveRDS(vhat, '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.rds')

INFO: vhat_mle is completed.
INFO: vhat_mle output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.rds
INFO: Checking mash_1: Fit MASH mixture model (time estimate: <15min for 70K by 49 matrix)
HINT: singularity exec  /home/hs3163/.sos/singularity/library/containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/tmp_zgtt6ms/singularity_run_46274.R
library(mashr)
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
vhat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.rds')
U = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.prior.rds')
mash_data = mash_set_data(dat$random.b, Shat=dat$random.s, alpha=1, V=vhat, zero_Bhat_Shat_reset = 1E3)
saveRDS(mash(mash_data, Ulist = U, outputlevel = 1), '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.mash_model.rds')

INFO: mash_1 is completed.
INFO: mash_1 output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.mash_model.rds
INFO: Checking mash_2: Compute posterior for the "strong" set of data as in Urbut et al 2017. This is optional because most of the time we want to apply the MASH model learned on much larger data-set.
HINT: singularity exec  /home/hs3163/.sos/singularity/library/containers/stephenslab.sif Rscript /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/tmptjukqsji/singularity_run_46274.R
library(mashr)
dat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
vhat = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.rds')
mash_data = mash_set_data(dat$strong.b, Shat=dat$strong.s, alpha=1, V=vhat, zero_Bhat_Shat_reset = 1E3)
mash_model = readRDS('/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.mash_model.rds')
saveRDS(mash_compute_posterior_matrices(mash_model, mash_data), '/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.posterior.rds')

INFO: mash_2 is completed.
INFO: mash_2 output:   /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/MASH/MASH/Ast_Exc_Inh_Mic_OPC_Oli.EZ.posterior.rds
INFO: Workflow mash (ID=w00c9ae7542945735) is tested successfully with 9 completed steps.

[hsun3163]

Archived: this is by design.

---

The container misuse problem is seemly fixed by specifying the full path of container, but this issue remains

It is rather strange that after specifying the --vhat to use "mle" method. vhat_simple was still called

[gaow]

Does it have to do with the fact that we recently remount the production drive on the cluster to a different mounting point? Would it help to modify bashrc to include the new path?

[hsun3163]

Does it have to do with the fact that we recently remount the production drive on the cluster to a different mounting point? Would it help to modify bashrc to include the new path?

It is unlikely, because I have been using soft link to specify the containers but only in here problem occurs. I have somehow avoid it for now so I will let it be for the moment.