Open koetjen opened 3 years ago
Hi, thank you for this! I tried to reproduced the error using the newest version of this repo but I could not reproduce this so far. I see that this has something to do with the fact that scover.nf
expects a file ending with .tsv.gz
- perhaps the difference in --no-home --cleanenv
has changed this. (I will do some more digging and will reply here if I have found the cause of the problem)
And thank you for reporting the problem with --no-home --cleanenv
, I will add this to the documentation.
Would you be willing to share the recipe for the singularity build so that I can try to reproduce it in a dockerfile? Our LSF doesn't support singularity, so I'm stuck running everything locally unless I can create a docker version
In my attempts to troubleshoot, I'm getting this error:
singularity verify work/singularity/jacobhepkema-scover_singularity_python-latest.img FATAL: Failed to verify: work/singularity/jacobhepkema-scover_singularity_python-latest.img: failed to load SIF container file: invalid SIF file: Magic |sqs?E:?| want |SIF_MAGIC|
This also happens when I pull directly outside of nextflow:
singularity pull shub://jacobhepkema/scover_singularity_python:latest INFO: Downloading shub image 3.18 GiB / 3.18 GiB [===================================================================] 100.00% 14.23 MiB/s 3m48s singularity verify scover_singularity_python_latest.sif FATAL: Failed to verify: scover_singularity_python_latest.sif: failed to load SIF container file: invalid SIF file: Magic |sqs?E:?| want |SIF_MAGIC|
Hi - yes, I'm happy to share the Singularity builds - I use different Singularity images for the different steps in the Nextflow workflow (it's not the most elegant solution):
Re: singularity, that's strange - I'll try this myself too to see if singularity verify
also fails for me
I think verify is failing for me because it's not compatible with the limited version of singularity I'm able to install locally (https://sylabs.io/singularity-desktop-macos/). If I can get some time available I will see if I can run some of the steps in docker. That would give me more flexibility on my end, and would allow other people at our institution to use it as well.
I've just installed singularity and nextflow, and running into difficulties executing the example data set, caused by: Missing output file(s)
*.tsv.gz
expected by processscover
singularity version 3.3.0-rc.1.658.g7427b73f1.dirty I needed to remove --no-home --cleanenv from the .conf file because those flags aren't available for exec in this version
nextflow run -profile local_cpu scover.nf --data data/example/small_dataset.tsv --celldata data/example/small_dataset_colData.tsv --name example_run --epochs 8 --num_candidates 8 --num_calibrations 4 --tomtom resources/Mus_musculus.meme N E X T F L O W ~ version 20.10.0 Launching
scover.nf
[tender_visvesvaraya] - revision: 025a058da8=========================================================================
scover v0.3 (Nov 27 2020)
=========================================================================
run name : example_run data path : data/example/small_dataset.tsv cell label data path : data/example/small_dataset_colData.tsv motif length : 12 amount of motifs : 300 epochs : 8 batch size : 128 K in K-fold CV : 10 number of cal : 4 number of candidates : 8 tomtom db file : resources/Mus_musculus.meme
random seed : 42
=========================================================================
executor > local (1) [a9/133309] process > scover [ 0%] 0 of 1 [- ] process > tomtom - [- ] process > motif_analysis - executor > local (1) [a9/133309] process > scover [100%] 1 of 1, failed: 1 ✘ [- ] process > tomtom - [- ] process > motif_analysis - Error executing process > 'scover'
Caused by: Missing output file(s)
*.tsv.gz
expected by processscover
Command executed:
scover.py /Users/karolynoetjen/scover/data/example/small_dataset.tsv /Users/karolynoetjen/scover/data/example/small_dataset_colData.tsv -name example_run -epochs 8 -num_candidates 8 -c 4 -v 10 -batch_size 128 -m 12 -d 300 -seed 42 -opt SGD -sigma_motifs_min 1E-7 -sigma_motifs_max 0.001 -sigma_net_min 0.00001 -sigma_net_max 0.01 -epsilon_min 0.0005 -epsilon_max 0.05
Command exit status: 0
Command output: (empty)