search

gustaveroussy / sopa

Technology-invariant pipeline for spatial omics analysis (Xenium / Visium HD / MERSCOPE / CosMx / PhenoCycler / MACSima / ...) that scales to millions of cells
https://gustaveroussy.github.io/sopa/
BSD 3-Clause "New" or "Revised" License
123 stars 14 forks source link

snakemake executes wrong rule #84

Open xiao233333 opened 3 months ago

xiao233333 commented 3 months ago

Thanks for your effort in providing this amazing tool. I encountered an issue while running the Snakemake pipeline for Cellpose segmentation on Phenocycler data. Could you please help me figure out what I did wrong here? Basically, I use the SLURM job system and the following command:

snakemake --profile slurm --config data_path=./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.qptiff --configfile=config/phenocycler/base_20X.yaml

It works fine until the step sopa resolve cellpose. 

    shell:

        sopa resolve cellpose ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.zarr --patch-dir ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.zarr/.sopa_cache/cellpose_boundaries

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Submitted batch job 9867213

Error executing rule resolve_cellpose on cluster (jobid: 2, external: Submitted batch job 9867213, jobscript: .snakemake/tmp.28g0oox_/snakejob.resolve_cellpose.2.sh). For error details see the cluster log and the log files of the involved rule(s).
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-07-02T180930.594385.snakemake.log

When I looked at the error file, it seems that it runs the sopa read again. 

AssertionError: Zarr directory 
./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.zarr already exists. Sopa will not 
continue to avoid overwritting files.
[Tue Jul  2 20:32:19 2024]
Error in rule to_spatialdata:
    jobid: 2
    input: ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.qptiff
    output: ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.zarr/.zgroup
    conda-env: spatialdata
    shell:

        sopa read ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.qptiff --sdata-path ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.zarr --technology "phenocycler"

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
SpatialData object path set to default: ./A2303-0606_MMT-01A-S05_nm_20240214_Scan1.zarr
To change this behavior, provide `--config sdata_path=...` when running the snakemake pipeline

Obviously, it will throw an error as there is already a .zarr folder there. I also noticed that the .zgroup file seems to have been deleted. I'm not sure what happened there.

quentinblampey commented 3 months ago

Hello @xiao233333, thanks for reporting this. I already experienced this issue, but I don't really understand what is happening. I currently believe that it's a strange Snakemake issue, which triggers the right rule, but tries to execute the command of another rule. Also, for now, it only happened on a Slurm cluster.

I wasn't able to have a clean reproducible example. Actually it seems that this happens almost randomly (even though rarely)...

Can you try to clear the snakemake cache, then clear the .zarr directory, and re-run the pipeline from zero? Sorry for the inconvenience If you have a reproducible example (for example using the toy dataset) I'll be happy to dive deeper!