search

gustaveroussy / sopa

Technology-invariant pipeline for spatial omics analysis (Xenium / Visium HD / MERSCOPE / CosMx / PhenoCycler / MACSima / ...) that scales to millions of cells
https://gustaveroussy.github.io/sopa/
BSD 3-Clause "New" or "Revised" License
123 stars 14 forks source link

sdata object has no cellpose or baysor boundaries #64

Open altunbulakli opened 4 months ago

altunbulakli commented 4 months ago

Hello to the SOPA creators.

I am quite a beginner to Python and the Command Line Interface. I have attempted to run the Ready made Snakemake pipeline for Xenium data (with the Baysor cell segmentation feature)

However, i get the following warning and the error in the resolve_baysor section of the pipeline:

[WARNING] (sopa._sdata) sdata object has no cellpose boundaries and no baysor boundaries. Consider running segmentation first.

and the error:

Finished job 3. 3 of 8 steps (38%) done MissingInputException in rule resolve_baysor in file C:\Windows\System32\sopa\workflow\Snakefile, line 125: Missing input files for rule resolve_baysor: output: C:/xenium_run/TC_070.zarr/.sopa_cache/baysor_boundaries_done, C:/xenium_run/TC_070.zarr/.sopa_cache/table affected files:

I was thinking that the Snakemake pipeline would run Baysor segmentation automatically. I also get a similar error when i try a Cellpose only config file or a Baysor.only config file. What could be the solution here? I have installed both the cellpose package (as part of the sopa installation) and the Baysor package according to their instructions.

Thank you very much for your help :)

quentinblampey commented 4 months ago

Hello @altunbulakli,

Indeed Sopa should run baysor, but in your case it seems it didn't. Can you show me the full log?

You should have a text file called TC_070.zarr/.sopa_cache/patches_file_baysor, can you show me how it looks? It should contain a list of patch indices. Maybe it is empty for some reasons

altunbulakli commented 4 months ago

Hey Quentin,

Thank you very much for your help! The patches_file_baysor looks like this, Indeed it is empty.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 21 22 23 24

quentinblampey commented 4 months ago

Actually, it looks as expected (it just a list of patches IDs). Each ID corresponds to the name of a directory inside TC_070.zarr/.sopa_cache/baysor_boundaries, do you have such directories?

Also, have you tried the toy example? Do you have the same issue?

altunbulakli commented 4 months ago

Sorry for the late reply. I wanted to make sure that Baysor was installed correctly and Cellpose was working outside of the Snakemake pipeline:

However, both the Baysor only config, Cellpose only config and the Toy Example command give a similar error: Here you can find the Toy example log:

(sopa) C:\Users\altun\Documents\sopa\workflow>snakemake --config sdata_path=tuto.zarr --configfile=config/toy/uniform_cellpose.yaml --cores 1 --use-conda Building DAG of jobs... Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count

aggregate 1 all 1 annotate 1 explorer 1 image_write 1 patchify_cellpose 1 report 1 resolve_cellpose 1 to_spatialdata 1 total 9

Select jobs to execute...

[Thu May 30 11:52:45 2024] rule to_spatialdata: output: tuto.zarr/.zgroup jobid: 5 reason: Missing output files: tuto.zarr/.zgroup resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=128000, mem_mib=122071

Activating conda environment: sopa [INFO] (sopa.utils.data) Image of size ((4, 2048, 2048)) with 400 cells and 100 transcripts per cell [INFO] (sopa.io.standardize) Writing the following spatialdata object to tuto.zarr: SpatialData object with: ├── Images │ └── 'image': SpatialImage[cyx] (4, 2048, 2048) ├── Points │ └── 'transcripts': DataFrame with shape: (, 4) (3D points) └── Shapes └── 'cells': GeoDataFrame shape: (400, 1) (2D shapes) with coordinate systems: ▸ 'global', with elements: image (Images), transcripts (Points), cells (Shapes) ▸ 'microns', with elements: transcripts (Points) [Thu May 30 11:52:55 2024] Finished job 5. 1 of 9 steps (11%) done Select jobs to execute...

[Thu May 30 11:52:55 2024] rule image_write: input: tuto.zarr/.zgroup output: tuto.explorer/morphology.ome.tif jobid: 7 reason: Missing output files: tuto.explorer/morphology.ome.tif; Input files updated by another job: tuto.zarr/.zgroup resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=64000, mem_mib=61036, partition=longq

Activating conda environment: sopa [WARNING] (sopa._sdata) sdata object has no cellpose boundaries and no baysor boundaries. Consider running segmentation first. [INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible) [INFO] (sopa.io.explorer.images) (Loading image of shape (4, 2048, 2048)) in memory [INFO] (sopa.io.explorer.images) > Image of shape (4, 2048, 2048) [INFO] (sopa.io.explorer.images) > Image of shape (4, 1024, 1024) [INFO] (sopa.io.explorer.images) > Image of shape (4, 512, 512) [INFO] (sopa.io.explorer.images) > Image of shape (4, 256, 256) [INFO] (sopa.io.explorer.images) > Image of shape (4, 128, 128) [INFO] (sopa.io.explorer.images) > Image of shape (4, 64, 64) [INFO] (sopa.io.explorer.converter) Saved files in the following directory: tuto.explorer [INFO] (sopa.io.explorer.converter) You can open the experiment with 'open tuto.explorer\experiment.xenium' [Thu May 30 11:53:01 2024] Finished job 7. 2 of 9 steps (22%) done Select jobs to execute...

[Thu May 30 11:53:01 2024] checkpoint patchify_cellpose: input: tuto.zarr/.zgroup output: tuto.zarr/.sopa_cache/patches_file_image, tuto.zarr/.sopa_cache/patches jobid: 4 reason: Missing output files: tuto.zarr/.sopa_cache/patches_file_image; Input files updated by another job: tuto.zarr/.zgroup resources: tmpdir=C:\Users\altun\AppData\Local\Temp DAG of jobs will be updated after completion.

Activating conda environment: sopa [INFO] (sopa.patches.patches) 4 patches were saved in sdata['sopa_patches'] Touching output file tuto.zarr/.sopa_cache/patches. [Thu May 30 11:53:07 2024] Finished job 4. 3 of 9 steps (33%) done MissingInputException in rule resolve_cellpose in file C:\Users\altun\Documents\sopa\workflow\Snakefile, line 113: Missing input files for rule resolve_cellpose: output: tuto.zarr/.sopa_cache/cellpose_boundaries_done affected files: tuto.zarr.sopa_cache\cellpose_boundaries\3.parquet tuto.zarr.sopa_cache\cellpose_boundaries\1.parquet tuto.zarr.sopa_cache\cellpose_boundaries\2.parquet tuto.zarr.sopa_cache\cellpose_boundaries\0.parquet

altunbulakli commented 4 months ago

And here are my logs for a Baysor only config file snakemake run:

(sopa) C:\Users\altun\Documents\sopa\workflow>snakemake --config data_path=C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070 --configfile=config/xenium/baysor_new.yaml --cores 1 --use-conda SpatialData object path set to default: C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr To change this behavior, provide --config sdata_path=... when running the snakemake pipeline Building DAG of jobs... Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count

aggregate 1 all 1 explorer 1 image_write 1 patchify_baysor 1 report 1 resolve_baysor 1 to_spatialdata 1 total 8

Select jobs to execute...

[Thu May 30 11:57:56 2024] rule to_spatialdata: input: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070 output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup jobid: 4 reason: Missing output files: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=128000, mem_mib=122071

Activating conda environment: sopa [INFO] (sopa.io.standardize) Writing the following spatialdata object to C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr: SpatialData object with: ├── Images │ └── 'morphology_focus': MultiscaleSpatialImage[cyx] (1, 20598, 22888), (1, 10299, 11444), (1, 5149, 5722), (1, 2574, 2861), (1, 1287, 1430) └── Points └── 'transcripts': DataFrame with shape: (, 10) (3D points) with coordinate systems: ▸ 'global', with elements: morphology_focus (Images), transcripts (Points) [Thu May 30 11:58:49 2024] Finished job 4. 1 of 8 steps (12%) done Select jobs to execute...

[Thu May 30 11:58:49 2024] rule image_write: input: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.explorer/morphology.ome.tif jobid: 6 reason: Missing output files: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.explorer/morphology.ome.tif; Input files updated by another job: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=64000, mem_mib=61036, partition=longq

Activating conda environment: sopa [WARNING] (sopa._sdata) sdata object has no cellpose boundaries and no baysor boundaries. Consider running segmentation first. [INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible) [INFO] (sopa.io.explorer.images) (Loading image of shape (1, 20598, 22888)) in memory [INFO] (sopa.io.explorer.images) > Image of shape (1, 20598, 22888) [INFO] (sopa.io.explorer.images) > Image of shape (1, 10299, 11444) [INFO] (sopa.io.explorer.images) > Image of shape (1, 5149, 5722) [INFO] (sopa.io.explorer.images) > Image of shape (1, 2574, 2861) [INFO] (sopa.io.explorer.images) > Image of shape (1, 1287, 1430) [INFO] (sopa.io.explorer.images) > Image of shape (1, 643, 715) [INFO] (sopa.io.explorer.converter) Saved files in the following directory: C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.explorer [INFO] (sopa.io.explorer.converter) You can open the experiment with 'open C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.explorer\experiment.xenium' [Thu May 30 11:59:12 2024] Finished job 6. 2 of 8 steps (25%) done Select jobs to execute...

[Thu May 30 11:59:12 2024] checkpoint patchify_baysor: input: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/patches_file_baysor, C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/baysor_boundaries jobid: 3 reason: Missing output files: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/patches_file_baysor; Input files updated by another job: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup resources: tmpdir=C:\Users\altun\AppData\Local\Temp DAG of jobs will be updated after completion.

Activating conda environment: sopa [INFO] (sopa.patches.patches) Writing sub-CSV for baysor [########################################] | 100% Completed | 56.31 s [INFO] (sopa.patches.patches) Patches saved in directory C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries [INFO] (sopa.patches.patches) Patch 0 has < 4000 transcripts. Baysor will not be run on it. [INFO] (sopa.patches.patches) Patch 20 has < 4000 transcripts. Baysor will not be run on it. [Thu May 30 12:00:16 2024] Finished job 3. 3 of 8 steps (38%) done MissingInputException in rule resolve_baysor in file C:\Users\altun\Documents\sopa\workflow\Snakefile, line 125: Missing input files for rule resolve_baysor: output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/baysor_boundaries_done, C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/table affected files: C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\21\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\17\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\4\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\24\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\16\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\18\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\11\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\23\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\1\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\13\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\18\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\16\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\24\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\14\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\13\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\11\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\19\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\10\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\7\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\15\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\15\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\1\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\8\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\17\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\5\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\8\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\23\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\22\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\12\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\3\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\4\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\19\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\2\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\12\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\9\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\9\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\6\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\3\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\6\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\5\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\10\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\21\segmentation_counts.loom C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\22\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\2\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\7\segmentation_polygons.json C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\14\segmentation_polygons.json

altunbulakli commented 4 months ago

At the end of this run, we have a C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries folder with the 0 to 24 patches.

Here is one of the folder's contents transcripts.csv

quentinblampey commented 4 months ago

Weirdly enough, it seems that snakemake uses an incorrect path. Indeed, in your logs I see tuto.zarr.sopa_cache\cellpose_boundaries\3.parquet, but there is a missing \, since it should be tuto.zarr\.sopa_cache\cellpose_boundaries\3.parquet

I don't understand why this happens, and all the other paths look correct, for instance tuto.zarr/.sopa_cache/patches

Just to make sure this is related to Snakemake, can you try to use the CLI as in this tutorial?

altunbulakli commented 4 months ago

Yes! I was able to follow the CLI tutorial, and it seems to segment the cells, resolve them and aggregate them successfully. Apart from a small error in report.html writing. I can also visualize the tutorial in Xenium explorer.

So it seems like the Snakemake pipeline that i have (I didnt modify the Snakefile so far) seems to have an error. One step closer :)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa read . --sdata-path tuto.zarr --technology uniform [INFO] (sopa.utils.data) Image of size ((4, 2048, 2048)) with 400 cells and 100 transcripts per cell [INFO] (sopa.io.standardize) Writing the following spatialdata object to tuto.zarr: SpatialData object with: ├── Images │ └── 'image': SpatialImage[cyx] (4, 2048, 2048) ├── Points │ └── 'transcripts': DataFrame with shape: (, 4) (3D points) └── Shapes └── 'cells': GeoDataFrame shape: (400, 1) (2D shapes) with coordinate systems: ▸ 'global', with elements: image (Images), transcripts (Points), cells (Shapes) ▸ 'microns', with elements: transcripts (Points)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa patchify image tuto.zarr --patch-width-pixel 1500 --patch-overlap-pixel 50 [INFO] (sopa.patches.patches) 4 patches were saved in sdata['sopa_patches']

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 0 [INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 0.93% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 1 [INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 0.00% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 2 [INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 2.30% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 3 [INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 0.00% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa resolve cellpose tuto.zarr Reading patches: 100%|███████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 82.79it/s] [INFO] (sopa.segmentation.stainings) Found 390 total cells Resolving conflicts: 100%|███████████████████████████████████████████████████████████| 92/92 [00:00<00:00, 4842.21it/s] [INFO] (sopa.segmentation.stainings) Added 370 cell boundaries in sdata['cellpose_boundaries']

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa aggregate tuto.zarr --gene-column genes --average-intensities --min-transcripts 10 [INFO] (sopa.segmentation.aggregate) Aggregating transcripts over 370 cells [########################################] | 100% Completed | 103.88 ms [INFO] (sopa.segmentation.aggregate) Filtering 0 cells C:\ProgramData\miniconda3\envs\sopa\lib\site-packages\spatialdata_core_elements.py:92: UserWarning: Key cellpose_boundaries already exists. Overwriting it. self._check_key(key, self.keys(), self._shared_keys) [INFO] (sopa.segmentation.aggregate) Averaging channels intensity over 370 cells with expansion 0.0 [########################################] | 100% Completed | 211.07 ms C:\Users\altun\Documents\sopa\sopa\segmentation\aggregate.py:268: ImplicitModificationWarning: Setting element .obsm['intensities'] of view, initializing view as actual. self.table.obsm[SopaKeys.INTENSITIES_OBSM] = pd.DataFrame( C:\ProgramData\miniconda3\envs\sopa\lib\site-packages\spatialdata_core_elements.py:92: UserWarning: Key cellpose_boundaries already exists. Overwriting it. self._check_key(key, self.keys(), self._shared_keys)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa report tuto.zarr report.html [INFO] (sopa.io.report.generate) Writing general_section [INFO] (sopa.io.report.generate) Writing cell_section [INFO] (sopa.io.report.generate) Writing channel_section [INFO] (sopa.io.report.generate) Writing transcripts_section [INFO] (sopa.io.report.generate) Writing representation_section [INFO] (sopa.io.report.generate) Computing UMAP on 370 cells [INFO] (sopa.io.report.generate) Writing report to report.html ┌─────────────────────────────── Traceback (most recent call last) ────────────────────────────────┐ │ C:\Users\altun\Documents\sopa\sopa\cli\app.py:212 in report │ │ │ │ 209 │ │ │ 210 │ sdata = read_zarr_standardized(sdata_path) │ │ 211 │ │ │ > 212 │ write_report(path, sdata) │ │ 213 │ │ │ │ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │ │ │ path = 'report.html' │ │ │ │ read_zarr_standardized = <function read_zarr_standardized at 0x000001BC80AC77F0> │ │ │ │ sdata = SpatialData object with: │ │ │ │ ├── Images │ │ │ │ │ └── 'image': SpatialImage[cyx] (4, 2048, 2048) │ │ │ │ ├── Points │ │ │ │ │ └── 'transcripts': DataFrame with shape: (40000, 4) (3D │ │ │ │ points) │ │ │ │ ├── Shapes │ │ │ │ │ ├── 'cellpose_boundaries': GeoDataFrame shape: (370, 1) (2D │ │ │ │ shapes) │ │ │ │ │ ├── 'cells': GeoDataFrame shape: (400, 1) (2D shapes) │ │ │ │ │ └── 'sopa_patches': GeoDataFrame shape: (4, 1) (2D shapes) │ │ │ │ └── Tables │ │ │ │ │ └── 'table': AnnData (370, 5) │ │ │ │ with coordinate systems: │ │ │ │ ▸ 'global', with elements: │ │ │ │ │ │ image (Images), transcripts (Points), cellpose_boundaries │ │ │ │ (Shapes), cells (Shapes), sopa_patches (Shapes) │ │ │ │ ▸ 'microns', with elements: │ │ │ │ │ │ transcripts (Points) │ │ │ │ sdata_path = 'tuto.zarr' │ │ │ │ write_report = <function write_report at 0x000001BC852F2A70> │ │ │ └──────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ C:\Users\altun\Documents\sopa\sopa\io\report\generate.py:47 in write_report │ │ │ │ 44 │ sections = SectionBuilder(sdata).compute_sections() │ │ 45 │ │ │ 46 │ log.info(f"Writing report to {path}") │ │ > 47 │ Root(sections).write(path) │ │ 48 │ │ 49 │ │ 50 def _kdeplot_vmax_quantile(values: np.ndarray, quantile: float = 0.95): │ │ │ │ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │ │ │ path = 'report.html' │ │ │ │ sdata = SpatialData object with: │ │ │ │ ├── Images │ │ │ │ │ └── 'image': SpatialImage[cyx] (4, 2048, 2048) │ │ │ │ ├── Points │ │ │ │ │ └── 'transcripts': DataFrame with shape: (40000, 4) (3D points) │ │ │ │ ├── Shapes │ │ │ │ │ ├── 'cellpose_boundaries': GeoDataFrame shape: (370, 1) (2D shapes) │ │ │ │ │ ├── 'cells': GeoDataFrame shape: (400, 1) (2D shapes) │ │ │ │ │ └── 'sopa_patches': GeoDataFrame shape: (4, 1) (2D shapes) │ │ │ │ └── Tables │ │ │ │ │ └── 'table': AnnData (370, 5) │ │ │ │ with coordinate systems: │ │ │ │ ▸ 'global', with elements: │ │ │ │ │ │ image (Images), transcripts (Points), cellpose_boundaries (Shapes), cells │ │ │ │ (Shapes), sopa_patches (Shapes) │ │ │ │ ▸ 'microns', with elements: │ │ │ │ │ │ transcripts (Points) │ │ │ │ sections = [ │ │ │ │ │ <sopa.io.report.engine.Section object at 0x000001BC857897E0>, │ │ │ │ │ <sopa.io.report.engine.Section object at 0x000001BC879D76A0>, │ │ │ │ │ <sopa.io.report.engine.Section object at 0x000001BC8A53E170>, │ │ │ │ │ <sopa.io.report.engine.Section object at 0x000001BC8A8573A0>, │ │ │ │ │ <sopa.io.report.engine.Section object at 0x000001BC8EE92740> │ │ │ │ ] │ │ │ └──────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ C:\Users\altun\Documents\sopa\sopa\io\report\engine.py:264 in write │ │ │ │ 261 │ │ self.sanity_check() │ │ 262 │ │ │ │ 263 │ │ with open(path, "w") as f: │ │ > 264 │ │ │ f.write(str(self)) │ │ 265 │ │ │ 266 │ def str(self) -> str: │ │ 267 │ │ return f""" │ │ │ │ ┌───────────────────────────────── locals ─────────────────────────────────┐ │ │ │ f = <_io.TextIOWrapper name='report.html' mode='w' encoding='cp1252'> │ │ │ │ path = 'report.html' │ │ │ │ self = <sopa.io.report.engine.Root object at 0x000001BC85623400> │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ │ │ │ C:\ProgramData\miniconda3\envs\sopa\lib\encodings\cp1252.py:19 in encode │ │ │ │ 16 │ │ 17 class IncrementalEncoder(codecs.IncrementalEncoder): │ │ 18 │ def encode(self, input, final=False): │ │ > 19 │ │ return codecs.charmap_encode(input,self.errors,encoding_table)[0] │ │ 20 │ │ 21 class IncrementalDecoder(codecs.IncrementalDecoder): │ │ 22 │ def decode(self, input, final=False): │ │ │ │ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │ │ │ final = False │ │ │ │ input = '\r\n <!DOCTYPE html>\r\n \r\n \r\n <meta │ │ │ │ charset="utf-8" />\r'+342740 │ │ │ │ self = <encodings.cp1252.IncrementalEncoder object at 0x000001BC8EEAFE80> │ │ │ └──────────────────────────────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────────────────────────────┘ UnicodeEncodeError: 'charmap' codec can't encode characters in position 209553-209555: character maps to

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa explorer write tuto.zarr --gene-column genes [INFO] (sopa.io.explorer.table) Writing table with 5 columns [INFO] (sopa.io.explorer.table) Writing 2 cell categories: region, slide [INFO] (sopa.io.explorer.shapes) Writing 370 cell polygons [INFO] (sopa.io.explorer.points) Writing 40000 transcripts [INFO] (sopa.io.explorer.points) > Level 0: 40000 transcripts [INFO] (sopa.io.explorer.points) > Level 1: 10000 transcripts [INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible) [INFO] (sopa.io.explorer.images) (Loading image of shape (4, 2048, 2048)) in memory [INFO] (sopa.io.explorer.images) > Image of shape (4, 2048, 2048) [INFO] (sopa.io.explorer.images) > Image of shape (4, 1024, 1024) [INFO] (sopa.io.explorer.images) > Image of shape (4, 512, 512) [INFO] (sopa.io.explorer.images) > Image of shape (4, 256, 256) [INFO] (sopa.io.explorer.images) > Image of shape (4, 128, 128) [INFO] (sopa.io.explorer.images) > Image of shape (4, 64, 64) [INFO] (sopa.io.explorer.converter) Saved files in the following directory: tuto.explorer [INFO] (sopa.io.explorer.converter) You can open the experiment with 'open tuto.explorer\experiment.xenium'

quentinblampey commented 4 months ago

Good to hear that the CLI works. I changed something in the way to handle the paths in the snakemake pipeline. Maybe it will fix this "missing /" issue, but since I can't reproduce the issue, I'm not sure. This will be available in the next version of sopa!

Same for the report, I just changed the file encoding, should be fixed in the next release of sopa. I will let you know when it's released!

quentinblampey commented 3 months ago

I just released sopa==1.1.0, it should fix at least the encoding issue, can you check?

josenimo commented 2 months ago

Dear @quentinblampey and @altunbulakli,

I am receiving a similar error while trying to process a subset dataset.

Details: I am trying to run sopa through snakemake locally on a virtual windows machine. I have two mamba environments, one for snakemake, and one with sopa. From the snakemake env I run snakemake --config data_path="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.tif" --configfile="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\ometif.yaml" --cores 10 --use-conda

It loads the image, but at checkpoint patchify_cellpose it fails claiming that MissingInputException in rule resolve_cellpose in file C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\Snakefile, line 154: Missing input files for rule resolve_cellpose:

The snakefile remains unchanged from sopa repo.

Happy to provide more details if needed :)

config.yaml ``` read: technology: aicsimageio patchify: patch_width_pixel: 2000 patch_overlap_pixel: 100 segmentation: cellpose: diameter: 25 channels: ["DAPI_bg"] flow_threshold: 1 cellprob_threshold: -6 min_area: 300 model_type: "nuclei" clip_limit: 0.2 gaussian_sigma: 1 aggregate: average_intensities: true min_intensity_ratio: 0.1 expand_radius_ratio: 0.1 ```
StackTrace ```python (snakemake) C:\Users\jnimoca\Desktop\SOPA\sopa>snakemake --config data_path="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.tif" --configfile="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\ometif.yaml" --cores 10 --use-conda SpatialData object path set to default: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr To change this behavior, provide `--config sdata_path=...` when running the snakemake pipeline Building DAG of jobs... Provided cores: 10 Rules claiming more threads will be scaled down. Job stats: job count ----------------- ------- aggregate 1 all 1 explorer 1 image_write 1 patchify_cellpose 1 report 1 resolve_cellpose 1 to_spatialdata 1 total 8 Select jobs to execute... [Thu Jul 25 15:05:35 2024] rule to_spatialdata: input: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.tif output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup jobid: 4 reason: Missing output files: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup resources: tmpdir=C:\Users\jnimoca\AppData\Local\Temp\4, mem_mb=128000, mem_mib=122071 Activating conda environment: sopa [WARNING] (sopa.io.standardize) No transcripts found. Some tools from sopa will not be available. [INFO] (sopa.io.standardize) Writing the following spatialdata object to C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr: SpatialData object └── Images └── 'image': DataTree[cyx] (15, 8000, 8000), (15, 4000, 4000), (15, 2000, 2000), (15, 1000, 1000), (15, 500, 500) with coordinate systems: ▸ 'global', with elements: image (Images) INFO The Zarr backing store has been changed from None the new file path: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr [Thu Jul 25 15:07:44 2024] Finished job 4. 1 of 8 steps (12%) done Select jobs to execute... [Thu Jul 25 15:07:44 2024] rule image_write: input: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.explorer/morphology.ome.tif jobid: 6 reason: Missing output files: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.explorer/morphology.ome.tif; Input files updated by another job: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup resources: tmpdir=C:\Users\jnimoca\AppData\Local\Temp\4, mem_mb=64000, mem_mib=61036, partition=longq Activating conda environment: sopa [Thu Jul 25 15:07:44 2024] checkpoint patchify_cellpose: input: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches_file_image, C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches jobid: 3 reason: Missing output files: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches_file_image; Input files updated by another job: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup resources: tmpdir=C:\Users\jnimoca\AppData\Local\Temp\4 DAG of jobs will be updated after completion. Activating conda environment: sopa [WARNING] (sopa._sdata) sdata object has no valid segmentation boundary. Consider running Sopa segmentation first. [INFO] (sopa.patches.patches) 25 patches were saved in sdata['sopa_patches'] [INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible) [INFO] (sopa.io.explorer.images) (Loading image of shape (15, 8000, 8000)) in memory Touching output file C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches. [Thu Jul 25 15:07:56 2024] Finished job 3. 2 of 8 steps (25%) done MissingInputException in rule resolve_cellpose in file C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\Snakefile, line 154: Missing input files for rule resolve_cellpose: output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/cellpose_boundaries_done affected files: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\10.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\15.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\22.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\20.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\6.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\19.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\8.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\16.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\13.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\24.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\0.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\14.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\3.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\21.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\18.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\1.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\2.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\11.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\9.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\4.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\5.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\23.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\12.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\7.parquet C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\17.parquet [INFO] (sopa.io.explorer.images) > Image of shape (15, 8000, 8000) [INFO] (sopa.io.explorer.images) > Image of shape (15, 4000, 4000) [INFO] (sopa.io.explorer.images) > Image of shape (15, 2000, 2000) [INFO] (sopa.io.explorer.images) > Image of shape (15, 1000, 1000) [INFO] (sopa.io.explorer.images) > Image of shape (15, 500, 500) [INFO] (sopa.io.explorer.images) > Image of shape (15, 250, 250) [INFO] (sopa.io.explorer.converter) Saved files in the following directory: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.explorer [INFO] (sopa.io.explorer.converter) You can open the experiment with 'open C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.explorer\experiment.xenium' ```
sopa mamba list ```python (sopa) C:\Users\jnimoca\Desktop\SOPA\sopa>mamba list # packages in environment at C:\Users\jnimoca\AppData\Local\miniforge3\envs\sopa: # # Name Version Build Channel aicsimageio 4.14.0 pypi_0 pypi aiobotocore 2.11.2 pypi_0 pypi aiohttp 3.9.5 pypi_0 pypi aioitertools 0.11.0 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi anndata 0.10.8 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi array-api-compat 1.7.1 pypi_0 pypi asciitree 0.3.3 pypi_0 pypi async-timeout 4.0.3 pypi_0 pypi attrs 23.2.0 pypi_0 pypi botocore 1.34.19 pypi_0 pypi bzip2 1.0.8 h2466b09_7 conda-forge ca-certificates 2024.7.4 h56e8100_0 conda-forge cellpose 3.0.10 pypi_0 pypi certifi 2024.7.4 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi click 8.1.7 pypi_0 pypi cloudpickle 3.0.0 pypi_0 pypi colorama 0.4.6 pypi_0 pypi colorcet 3.1.0 pypi_0 pypi contourpy 1.2.1 pypi_0 pypi cycler 0.12.1 pypi_0 pypi dask 2024.7.1 pypi_0 pypi dask-expr 1.1.9 pypi_0 pypi dask-image 2024.5.3 pypi_0 pypi datashader 0.16.3 pypi_0 pypi distributed 2024.7.1 pypi_0 pypi elementpath 4.4.0 pypi_0 pypi exceptiongroup 1.2.2 pypi_0 pypi fasteners 0.19 pypi_0 pypi fastremap 1.15.0 pypi_0 pypi fcsparser 0.2.8 pypi_0 pypi filelock 3.15.4 pypi_0 pypi flake8 7.1.0 pypi_0 pypi fonttools 4.53.1 pypi_0 pypi frozenlist 1.4.1 pypi_0 pypi fsspec 2023.6.0 pypi_0 pypi geopandas 1.0.1 pypi_0 pypi h5py 3.11.0 pypi_0 pypi idna 3.7 pypi_0 pypi imagecodecs 2024.6.1 pypi_0 pypi imageio 2.34.2 pypi_0 pypi importlib-metadata 8.0.0 pypi_0 pypi intel-openmp 2021.4.0 pypi_0 pypi jinja2 3.1.4 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi joblib 1.4.2 pypi_0 pypi kiwisolver 1.4.5 pypi_0 pypi lamin-utils 0.13.2 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi legacy-api-wrap 1.4 pypi_0 pypi libffi 3.4.2 h8ffe710_5 conda-forge libsqlite 3.46.0 h2466b09_0 conda-forge libzlib 1.3.1 h2466b09_1 conda-forge llvmlite 0.43.0 pypi_0 pypi locket 1.0.0 pypi_0 pypi lxml 4.9.4 pypi_0 pypi markdown-it-py 3.0.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib 3.9.1 pypi_0 pypi mccabe 0.7.0 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mkl 2021.4.0 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi msgpack 1.0.8 pypi_0 pypi multidict 6.0.5 pypi_0 pypi multipledispatch 1.0.0 pypi_0 pypi multiscale-spatial-image 1.0.0 pypi_0 pypi natsort 8.4.0 pypi_0 pypi networkx 3.3 pypi_0 pypi numba 0.60.0 pypi_0 pypi numcodecs 0.13.0 pypi_0 pypi numpy 1.26.4 pypi_0 pypi ome-types 0.5.1.post1 pypi_0 pypi ome-zarr 0.9.0 pypi_0 pypi opencv-python 4.10.0.84 pypi_0 pypi opencv-python-headless 4.10.0.84 pypi_0 pypi openssl 3.3.1 h2466b09_2 conda-forge packaging 24.1 pypi_0 pypi pandas 2.2.2 pypi_0 pypi param 2.1.1 pypi_0 pypi partd 1.4.2 pypi_0 pypi patsy 0.5.6 pypi_0 pypi pillow 10.4.0 pypi_0 pypi pims 0.7 pypi_0 pypi pip 24.0 pyhd8ed1ab_0 conda-forge platformdirs 4.2.2 pypi_0 pypi pooch 1.8.2 pypi_0 pypi psutil 6.0.0 pypi_0 pypi pyarrow 17.0.0 pypi_0 pypi pycodestyle 2.12.0 pypi_0 pypi pyct 0.5.0 pypi_0 pypi pydantic 2.8.2 pypi_0 pypi pydantic-compat 0.1.2 pypi_0 pypi pydantic-core 2.20.1 pypi_0 pypi pyflakes 3.2.0 pypi_0 pypi pygments 2.18.0 pypi_0 pypi pynndescent 0.5.13 pypi_0 pypi pyogrio 0.9.0 pypi_0 pypi pyparsing 3.1.2 pypi_0 pypi pyproj 3.6.1 pypi_0 pypi python 3.10.0 hcf16a7b_3_cpython conda-forge python-dateutil 2.9.0.post0 pypi_0 pypi pytz 2024.1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi readfcs 1.1.8 pypi_0 pypi requests 2.32.3 pypi_0 pypi resource-backed-dask-array 0.1.0 pypi_0 pypi rich 13.7.1 pypi_0 pypi roifile 2024.5.24 pypi_0 pypi s3fs 0.6.0 pypi_0 pypi scanpy 1.10.2 pypi_0 pypi scikit-image 0.24.0 pypi_0 pypi scikit-learn 1.5.1 pypi_0 pypi scipy 1.14.0 pypi_0 pypi seaborn 0.13.2 pypi_0 pypi session-info 1.0.0 pypi_0 pypi setuptools 71.0.4 pyhd8ed1ab_0 conda-forge shapely 2.0.5 pypi_0 pypi shellingham 1.5.4 pypi_0 pypi six 1.16.0 pypi_0 pypi slicerator 1.1.0 pypi_0 pypi sopa 1.1.1 pypi_0 pypi sortedcontainers 2.4.0 pypi_0 pypi spatial-image 1.1.0 pypi_0 pypi spatialdata 0.2.1 pypi_0 pypi spatialdata-io 0.1.3.post0 pypi_0 pypi sqlite 3.46.0 h2466b09_0 conda-forge statsmodels 0.14.2 pypi_0 pypi stdlib-list 0.10.0 pypi_0 pypi sympy 1.13.1 pypi_0 pypi tbb 2021.13.0 pypi_0 pypi tblib 3.0.0 pypi_0 pypi threadpoolctl 3.5.0 pypi_0 pypi tifffile 2023.2.28 pypi_0 pypi tk 8.6.13 h5226925_1 conda-forge toolz 0.12.1 pypi_0 pypi torch 2.3.1 pypi_0 pypi tornado 6.4.1 pypi_0 pypi tqdm 4.66.4 pypi_0 pypi typer 0.12.3 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2024.1 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_0 conda-forge umap-learn 0.5.6 pypi_0 pypi urllib3 2.0.7 pypi_0 pypi vc 14.3 h8a93ad2_20 conda-forge vc14_runtime 14.40.33810 ha82c5b3_20 conda-forge vs2015_runtime 14.40.33810 h3bf8584_20 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge wrapt 1.16.0 pypi_0 pypi xarray 2024.6.0 pypi_0 pypi xarray-dataclasses 1.8.0 pypi_0 pypi xarray-datatree 0.0.14 pypi_0 pypi xarray-schema 0.0.3 pypi_0 pypi xarray-spatial 0.4.0 pypi_0 pypi xmlschema 3.3.1 pypi_0 pypi xsdata 24.3.1 pypi_0 pypi xz 5.2.6 h8d14728_0 conda-forge yarl 1.9.4 pypi_0 pypi zarr 2.15.0 pypi_0 pypi zict 3.0.0 pypi_0 pypi zipp 3.19.2 pypi_0 pypi ```
snakemake mamba list ```python (snakemake) C:\Users\jnimoca\Desktop\SOPA\sopa>mamba list # packages in environment at C:\Users\jnimoca\AppData\Local\miniforge3\envs\snakemake: # # Name Version Build Channel aioeasywebdav 2.4.0 pyha770c72_0 conda-forge aiohttp 3.9.5 py310h8d17308_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge amply 0.1.6 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attmap 0.13.2 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pyh71513ae_0 conda-forge bcrypt 4.2.0 py310hc226416_0 conda-forge boto3 1.34.148 pyhd8ed1ab_0 conda-forge botocore 1.34.148 pyge310_1234567_0 conda-forge brotli-python 1.1.0 py310h00ffb61_1 conda-forge bzip2 1.0.8 h2466b09_7 conda-forge c-ares 1.32.3 h2466b09_0 conda-forge ca-certificates 2024.7.4 h56e8100_0 conda-forge cachetools 5.4.0 pyhd8ed1ab_0 conda-forge certifi 2024.7.4 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py310h8d17308_0 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge configargparse 1.7 pyhd8ed1ab_0 conda-forge connection_pool 0.0.3 pyhd3deb0d_0 conda-forge cryptography 42.0.8 py310h25a5809_0 conda-forge datrie 0.8.2 py310h8d17308_7 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge docutils 0.21.2 pyhd8ed1ab_0 conda-forge dpath 2.2.0 pyha770c72_0 conda-forge dropbox 12.0.2 pyhd8ed1ab_0 conda-forge eido 0.2.2 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.2 pyhd8ed1ab_0 conda-forge filechunkio 1.8 py_2 conda-forge frozenlist 1.4.1 py310h8d17308_0 conda-forge ftputil 5.1.0 pyhd8ed1ab_0 conda-forge gitdb 4.0.11 pyhd8ed1ab_0 conda-forge gitpython 3.1.43 pyhd8ed1ab_0 conda-forge glpk 5.0 h8ffe710_0 conda-forge google-api-core 2.19.1 pyhd8ed1ab_0 conda-forge google-api-python-client 2.137.0 pyhd8ed1ab_0 conda-forge google-auth 2.32.0 pyhff2d567_0 conda-forge google-auth-httplib2 0.2.0 pyhd8ed1ab_0 conda-forge google-cloud-core 2.4.1 pyhd8ed1ab_0 conda-forge google-cloud-storage 2.18.0 pyhff2d567_0 conda-forge google-crc32c 1.1.2 py310ha7be474_5 conda-forge google-resumable-media 2.7.0 pyhd8ed1ab_0 conda-forge googleapis-common-protos 1.63.2 pyhd8ed1ab_0 conda-forge grpcio 1.62.2 py310hb84602e_0 conda-forge h2 4.1.0 pyhd8ed1ab_0 conda-forge hpack 4.0.0 pyh9f0ad1d_0 conda-forge httplib2 0.22.0 pyhd8ed1ab_0 conda-forge humanfriendly 10.0 py310h5588dad_5 conda-forge hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge idna 3.7 pyhd8ed1ab_0 conda-forge importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge intel-openmp 2024.2.0 h57928b3_980 conda-forge jinja2 3.1.4 pyhd8ed1ab_0 conda-forge jmespath 1.0.1 pyhd8ed1ab_0 conda-forge jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.2 py310h5588dad_0 conda-forge libabseil 20240116.2 cxx17_he0c23c2_1 conda-forge libblas 3.9.0 23_win64_mkl conda-forge libcblas 3.9.0 23_win64_mkl conda-forge libcrc32c 1.1.2 h0e60522_0 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libgrpc 1.62.2 h5273850_0 conda-forge libhwloc 2.11.1 default_h8125262_1000 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge liblapack 3.9.0 23_win64_mkl conda-forge libprotobuf 4.25.3 h503648d_0 conda-forge libre2-11 2023.09.01 hf8d8778_2 conda-forge libsqlite 3.46.0 h2466b09_0 conda-forge libxml2 2.12.7 h0f24e4e_4 conda-forge libzlib 1.3.1 h2466b09_1 conda-forge logmuse 0.2.6 pyh8c360ce_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.5 py310h8d17308_0 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mkl 2024.1.0 h66d3029_694 conda-forge multidict 6.0.5 py310h8d17308_0 conda-forge nbformat 5.10.4 pyhd8ed1ab_0 conda-forge numpy 2.0.0 py310h1ec8c79_0 conda-forge oauth2client 4.1.3 py_0 conda-forge openssl 3.3.1 h2466b09_2 conda-forge packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.2.2 py310hb4db72f_1 conda-forge paramiko 3.4.0 pyhd8ed1ab_0 conda-forge peppy 0.40.4 pyhd8ed1ab_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge plac 1.4.3 pyhd8ed1ab_0 conda-forge platformdirs 4.2.2 pyhd8ed1ab_0 conda-forge pluggy 1.5.0 pyhd8ed1ab_0 conda-forge ply 3.11 pyhd8ed1ab_2 conda-forge prettytable 3.10.2 pyhd8ed1ab_0 conda-forge proto-plus 1.23.0 pyhd8ed1ab_0 conda-forge protobuf 4.25.3 py310hf536535_0 conda-forge psutil 6.0.0 py310ha8f682b_0 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge pulp 2.7.0 py310h5588dad_1 conda-forge pyasn1 0.6.0 pyhd8ed1ab_0 conda-forge pyasn1-modules 0.4.0 pyhd8ed1ab_0 conda-forge pycparser 2.22 pyhd8ed1ab_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pynacl 1.5.0 py310h635b8f1_3 conda-forge pyopenssl 24.2.1 pyhd8ed1ab_0 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pyreadline3 3.4.1 py310h5588dad_3 conda-forge pysftp 0.2.9 py_1 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge pytest 8.3.1 pyhd8ed1ab_0 conda-forge python 3.10.0 hcf16a7b_3_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.20.0 pyhd8ed1ab_0 conda-forge python-irodsclient 2.0.1 pyhd8ed1ab_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.10 4_cp310 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywin32 306 py310h00ffb61_2 conda-forge pyyaml 6.0.1 py310h8d17308_1 conda-forge re2 2023.09.01 hd3b24a8_2 conda-forge referencing 0.35.1 pyhd8ed1ab_0 conda-forge requests 2.32.3 pyhd8ed1ab_0 conda-forge reretry 0.11.8 pyhd8ed1ab_0 conda-forge rich 13.7.1 pyhd8ed1ab_0 conda-forge rpds-py 0.19.1 py310hc226416_0 conda-forge rsa 4.9 pyhd8ed1ab_0 conda-forge s3transfer 0.10.2 pyhd8ed1ab_0 conda-forge setuptools 71.0.4 pyhd8ed1ab_0 conda-forge setuptools-scm 8.1.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge slacker 0.14.0 py_0 conda-forge smart_open 7.0.4 pyhd8ed1ab_0 conda-forge smmap 5.0.0 pyhd8ed1ab_0 conda-forge snakemake 7.32.4 hdfd78af_1 bioconda snakemake-minimal 7.32.4 pyhdfd78af_1 bioconda sqlite 3.46.0 h2466b09_0 conda-forge stone 3.3.2 pyhd8ed1ab_0 conda-forge stopit 1.1.2 py_0 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tbb 2021.12.0 hc790b64_3 conda-forge throttler 1.2.2 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5226925_1 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toposort 1.10 pyhd8ed1ab_0 conda-forge traitlets 5.14.3 pyhd8ed1ab_0 conda-forge typing-extensions 4.12.2 hd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge ubiquerg 0.8.0 pyhd8ed1ab_0 conda-forge ucrt 10.0.22621.0 h57928b3_0 conda-forge uritemplate 4.1.1 pyhd8ed1ab_0 conda-forge urllib3 2.2.2 pyhd8ed1ab_1 conda-forge vc 14.3 h8a93ad2_20 conda-forge vc14_runtime 14.40.33810 ha82c5b3_20 conda-forge veracitools 0.1.3 py_0 conda-forge vs2015_runtime 14.40.33810 h3bf8584_20 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge wrapt 1.16.0 py310h8d17308_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge yarl 1.9.4 py310h8d17308_0 conda-forge yte 1.5.4 pyha770c72_0 conda-forge zipp 3.19.2 pyhd8ed1ab_0 conda-forge zstandard 0.23.0 py310he5e10e1_0 conda-forge zstd 1.5.6 h0ea2cb4_0 conda-forge ```
quentinblampey commented 2 months ago

Hello @josenimo,

Can you check if the "missing" files are really missing? E.g., does this file exist?

C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\10.parquet

If the file exists, then snakemake is not detecting it for some reasons. I suspect it has something to do with how paths are handled in Windows.

Two questions to check if my assumptions are correct:

josenimo commented 2 months ago

Hey @quentinblampey , thank you for your quick responses.

Update:

  1. Files are actually not there, only found patches and patches_file_image inside .sopa_cache
  2. toy example does not run, and fails in a similar way
  3. It runs perfectly on MacOS

Happy to provide more details to help solve Windows issues.

quentinblampey commented 2 months ago

Ok thanks for the details. I think it's an issue related to snakemake, which doesn't trigger the rules that create the input needed for the resolve_cellpose rule Can you tell me which version of snakemake and windows do you have?

When I have access to a windows laptop (i.e. not before late august), I'll check this and try to solve this!

Meanwhile, the CLI and the API should work fine on windows :)

josenimo commented 2 months ago

snakemake version 7.32.4 Windows Server 2022 Standard All good, thank you for the info. I almost got it running on SLURM :)

marsdenl commented 2 weeks ago

@quentinblampey @josenimo I have a similar issue on Windows also. .sopa_cache only contains patches and patches_file_image and the run fails after: Writing tiles: 100%|█████████████████████▉| 4796/4797 [07:28<00:00, 10.70it/s] [INFO] (sopa.io.explorer.images) (Loading image of shape (3, 20713, 19763)) in memory [INFO] (sopa.io.explorer.images) > Image of shape (3, 20713, 19763) [INFO] (sopa.io.explorer.images) > Image of shape (3, 10356, 9881) [INFO] (sopa.io.explorer.images) > Image of shape (3, 5178, 4940) [INFO] (sopa.io.explorer.images) > Image of shape (3, 2589, 2470) [INFO] (sopa.io.explorer.images) > Image of shape (3, 1294, 1235) [INFO] (sopa.io.explorer.converter) Saved files in the following directory: Z:\Queries\Data\Batch5_region0\region_0.explorer [INFO] (sopa.io.explorer.converter) You can open the experiment with 'open Z:\Queries\Data\Batch5_region0\region_0.explorer\experiment.xenium'

Let me know if anything comes of this! Would love to use snakemake to automate things a bit.

josenimo commented 2 weeks ago

Sorry @marsdenl, havent seen this before, can you share a bit more? you are running on snakemake right? default paramenters?

marsdenl commented 2 weeks ago

Hey @josenimo. Yes sorry for the very vague comment and thanks for taking the timer to answer. I've been trying to run it on snakemake yes with the following config: baysor_cellpose.yaml.txt

The output looks relatively fine for the steps completed:

SpatialData object path set to default: Z:\Queries\Data\Batch5_region0\region_0.zarr To change this behavior, provide --config sdata_path=... when running the snakemake pipeline Building DAG of jobs... Provided cores: 8 Rules claiming more threads will be scaled down. Job stats: job count

aggregate 1 all 1 explorer 1 image_write 1 patchify_baysor 1 patchify_cellpose 1 report 1 resolve_baysor 1 resolve_cellpose 1 to_spatialdata 1 total 10

Select jobs to execute...

[Mon Sep 16 13:28:41 2024] rule to_spatialdata: input: Z:/Queries/Data/Batch5_region0/region_0 output: Z:/Queries/Data/Batch5_region0/region_0.zarr/.zgroup jobid: 4 reason: Missing output files: Z:/Queries/Data/Batch5_region0/region_0.zarr/.zgroup resources: tmpdir=C:\Users\marsdenl\AppData\Local\Temp, mem_mb=128000, mem_mib=122071

Activating conda environment: sopa C:\Users\marsdenl\AppData\Local\miniconda3\envs\sopa\lib\functools.py:926: UserWarning: The index of the dataframe is not monotonic increasing. It is recommended to sort the data to adjust the order of the index before calling .parse() to avoid possible problems due to unknown divisions return method.get(obj, cls)(*args, **kwargs) INFO The column "global_x" has now been renamed to "x"; the column "x" was already present in the dataframe, and will be dropped. INFO The column "global_y" has now been renamed to "y"; the column "y" was already present in the dataframe, and will be dropped. [INFO] (sopa.io.standardize) Writing the following spatialdata object to Z:\Queries\Data\Batch5_region0\region_0.zarr: SpatialData object ├── Images │ └── 'Batch5_region0_region_0_z3': DataTree[cyx] (3, 41426, 39526), (3, 20713, 19763), (3, 10356, 9881), (3, 5178, 4940), (3, 2589, 2470) └── Points └── 'Batch5_region0_region_0_transcripts': DataFrame with shape: (, 9) (2D points) with coordinate systems: ▸ 'global', with elements: Batch5_region0_region_0_z3 (Images), Batch5_region0_region_0_transcripts (Points) INFO The Zarr backing store has been changed from None the new file path: Z:\Queries\Data\Batch5_region0\region_0.zarr [Mon Sep 16 13:34:29 2024] Finished job 4. 1 of 10 steps (10%) done Select jobs to execute...

[Mon Sep 16 13:34:29 2024] checkpoint patchify_cellpose: input: Z:/Queries/Data/Batch5_region0/region_0.zarr/.zgroup output: Z:/Queries/Data/Batch5_region0/region_0.zarr/.sopa_cache/patches_file_image, Z:/Queries/Data/Batch5_region0/region_0.zarr/.sopa_cache/patches jobid: 6 reason: Missing output files: Z:/Queries/Data/Batch5_region0/region_0.zarr/.sopa_cache/patches_file_image; Input files updated by another job: Z:/Queries/Data/Batch5_region0/region_0.zarr/.zgroup resources: tmpdir=C:\Users\marsdenl\AppData\Local\Temp DAG of jobs will be updated after completion.

Activating conda environment: sopa

[Mon Sep 16 13:34:29 2024] rule image_write: input: Z:/Queries/Data/Batch5_region0/region_0.zarr/.zgroup output: Z:/Queries/Data/Batch5_region0/region_0.explorer/morphology.ome.tif jobid: 8 reason: Missing output files: Z:/Queries/Data/Batch5_region0/region_0.explorer/morphology.ome.tif; Input files updated by another job: Z:/Queries/Data/Batch5_region0/region_0.zarr/.zgroup resources: tmpdir=C:\Users\marsdenl\AppData\Local\Temp, mem_mb=64000, mem_mib=61036, partition=longq

Activating conda environment: sopa [WARNING] (sopa._sdata) sdata object has no valid segmentation boundary. Consider running Sopa segmentation first. [INFO] (sopa.patches.patches) 56 patches were saved in sdata['sopa_patches'] [INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible) Writing tiles: 0%| | 0/4797 [00:00<?, ?it/s][INFO] (sopa.io.explorer.images) > Image of shape (3, 41426, 39526) Touching output file Z:/Queries/Data/Batch5_region0/region_0.zarr/.sopa_cache/patches. [Mon Sep 16 13:37:04 2024] Finished job 6. 2 of 10 steps (20%) done Writing tiles: 0%| | 1/4797 [00:00<22:50, 3.50it/s]MissingInputException in rule resolve_cellpose in file C:\Users\marsdenl\sopa\workflow\Snakefile, line 154: Missing input files for rule resolve_cellpose: output: Z:/Queries/Data/Batch5_region0/region_0.zarr/.sopa_cache/cellpose_boundaries_done affected files: Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\1.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\51.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\31.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\24.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\44.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\3.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\4.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\18.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\0.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\54.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\2.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\46.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\33.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\52.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\17.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\39.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\6.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\29.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\49.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\9.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\7.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\20.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\41.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\13.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\47.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\55.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\43.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\42.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\5.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\40.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\25.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\11.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\15.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\16.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\34.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\12.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\22.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\53.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\37.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\35.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\38.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\27.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\48.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\50.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\36.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\21.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\23.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\14.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\26.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\10.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\45.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\30.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\8.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\28.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\32.parquet Z:\Queries\Data\Batch5_region0\region_0.zarr.sopa_cache\cellpose_boundaries\19.parquet Writing tiles: 100%|███████████████████████████████████████████████████████████▉| 4796/4797 [06:28<00:00, 12.35it/s] [INFO] (sopa.io.explorer.images) (Loading image of shape (3, 20713, 19763)) in memory [INFO] (sopa.io.explorer.images) > Image of shape (3, 20713, 19763) [INFO] (sopa.io.explorer.images) > Image of shape (3, 10356, 9881) [INFO] (sopa.io.explorer.images) > Image of shape (3, 5178, 4940) [INFO] (sopa.io.explorer.images) > Image of shape (3, 2589, 2470) [INFO] (sopa.io.explorer.images) > Image of shape (3, 1294, 1235) [INFO] (sopa.io.explorer.converter) Saved files in the following directory: Z:\Queries\Data\Batch5_region0\region_0.explorer [INFO] (sopa.io.explorer.converter) You can open the experiment with 'open Z:\Queries\Data\Batch5_region0\region_0.explorer\experiment.xenium'

But there is no folder with all the patches, no segmentation output per say, no table directory with cell x gene matrices and the .explorer file is empty. Any idea why this might be the case?

Thank you for your help :)

quentinblampey commented 6 days ago

Hello all, Sorry for taking so long, I couldn't find a Windows that I could take for a few hours/days to reproduce the issue. One solution to move forward could be to have a short call with one of you so that we can debug in live. If you're interested, please send me an email at quentin.blampey@centralesupelec.fr!