search

jacobhepkema / scover

scover
MIT License
22 stars 0 forks source link

Error Missing output file(s) #1

Open koetjen opened 3 years ago

koetjen commented 3 years ago

I've just installed singularity and nextflow, and running into difficulties executing the example data set, caused by: Missing output file(s) *.tsv.gz expected by process scover

singularity version 3.3.0-rc.1.658.g7427b73f1.dirty I needed to remove --no-home --cleanenv from the .conf file because those flags aren't available for exec in this version

nextflow run -profile local_cpu scover.nf --data data/example/small_dataset.tsv --celldata data/example/small_dataset_colData.tsv --name example_run --epochs 8 --num_candidates 8 --num_calibrations 4 --tomtom resources/Mus_musculus.meme N E X T F L O W ~ version 20.10.0 Launching scover.nf [tender_visvesvaraya] - revision: 025a058da8

=========================================================================

scover v0.3 (Nov 27 2020)

=========================================================================

run name : example_run data path : data/example/small_dataset.tsv cell label data path : data/example/small_dataset_colData.tsv motif length : 12 amount of motifs : 300 epochs : 8 batch size : 128 K in K-fold CV : 10 number of cal : 4 number of candidates : 8 tomtom db file : resources/Mus_musculus.meme
random seed : 42

=========================================================================

executor > local (1) [a9/133309] process > scover [ 0%] 0 of 1 [- ] process > tomtom - [- ] process > motif_analysis - executor > local (1) [a9/133309] process > scover [100%] 1 of 1, failed: 1 ✘ [- ] process > tomtom - [- ] process > motif_analysis - Error executing process > 'scover'

Caused by: Missing output file(s) *.tsv.gz expected by process scover

Command executed:

scover.py /Users/karolynoetjen/scover/data/example/small_dataset.tsv /Users/karolynoetjen/scover/data/example/small_dataset_colData.tsv -name example_run -epochs 8 -num_candidates 8 -c 4 -v 10 -batch_size 128 -m 12 -d 300 -seed 42 -opt SGD -sigma_motifs_min 1E-7 -sigma_motifs_max 0.001 -sigma_net_min 0.00001 -sigma_net_max 0.01 -epsilon_min 0.0005 -epsilon_max 0.05

Command exit status: 0

Command output: (empty)

jacobhepkema commented 3 years ago

Hi, thank you for this! I tried to reproduced the error using the newest version of this repo but I could not reproduce this so far. I see that this has something to do with the fact that scover.nf expects a file ending with .tsv.gz- perhaps the difference in --no-home --cleanenv has changed this. (I will do some more digging and will reply here if I have found the cause of the problem)

And thank you for reporting the problem with --no-home --cleanenv, I will add this to the documentation.

koetjen commented 3 years ago

Would you be willing to share the recipe for the singularity build so that I can try to reproduce it in a dockerfile? Our LSF doesn't support singularity, so I'm stuck running everything locally unless I can create a docker version

koetjen commented 3 years ago

In my attempts to troubleshoot, I'm getting this error:

singularity verify work/singularity/jacobhepkema-scover_singularity_python-latest.img FATAL: Failed to verify: work/singularity/jacobhepkema-scover_singularity_python-latest.img: failed to load SIF container file: invalid SIF file: Magic |sqs?E:?| want |SIF_MAGIC|

This also happens when I pull directly outside of nextflow:

singularity pull shub://jacobhepkema/scover_singularity_python:latest INFO: Downloading shub image 3.18 GiB / 3.18 GiB [===================================================================] 100.00% 14.23 MiB/s 3m48s singularity verify scover_singularity_python_latest.sif FATAL: Failed to verify: scover_singularity_python_latest.sif: failed to load SIF container file: invalid SIF file: Magic |sqs?E:?| want |SIF_MAGIC|

jacobhepkema commented 3 years ago

Hi - yes, I'm happy to share the Singularity builds - I use different Singularity images for the different steps in the Nextflow workflow (it's not the most elegant solution):

Re: singularity, that's strange - I'll try this myself too to see if singularity verify also fails for me

koetjen commented 3 years ago

I think verify is failing for me because it's not compatible with the limited version of singularity I'm able to install locally (https://sylabs.io/singularity-desktop-macos/). If I can get some time available I will see if I can run some of the steps in docker. That would give me more flexibility on my end, and would allow other people at our institution to use it as well.