IndexError, trying MCMICRO TMA Cylinter run

[josenimo]

Dear @gjbaker ,

again thank you for the fantastic software. I am having a couple of problems running Cylinter.

My setup: Cylinter 0.0.47 installed in a mamba environment. MCMICRO folder structure present in a server drive, ran in with tma: true in the params.yml file

Command I am using: cylinter Cylinter_config.yml

copy of config file: Cylinter_config copy.txt

the files of all segmented cores are present in the dearray folder

Stack Trace:

``` (cylinter-env47) CMP06623:P21E05_HN26 jnimoca$ cylinter Cylinter_config.yml INFO: Reading configuration file INFO: Executing pipeline Running: INFO: ====================================================================== INFO: RUNNING MODULE: aggregateData Traceback (most recent call last): File "/opt/homebrew/Caskroom/mambaforge/base/envs/cylinter-env47/bin/cylinter", line 10, in sys.exit(main()) File "/opt/homebrew/Caskroom/mambaforge/base/envs/cylinter-env47/lib/python3.10/site-packages/cylinter/cylinter.py", line 49, in main pipeline.run_pipeline(config, args.module) File "/opt/homebrew/Caskroom/mambaforge/base/envs/cylinter-env47/lib/python3.10/site-packages/cylinter/pipeline.py", line 132, in run_pipeline data = module(data, qc, config) # getattr(qc, module) File "/opt/homebrew/Caskroom/mambaforge/base/envs/cylinter-env47/lib/python3.10/site-packages/cylinter/components.py", line 56, in wrapper result = func(*args, **kwargs) File "/opt/homebrew/Caskroom/mambaforge/base/envs/cylinter-env47/lib/python3.10/site-packages/cylinter/modules/aggregateData.py", line 14, in aggregateData check, markers_filepath = input_check(self) File "/opt/homebrew/Caskroom/mambaforge/base/envs/cylinter-env47/lib/python3.10/site-packages/cylinter/utils.py", line 86, in input_check segmentation_method = key.split('--')[1].split('_')[0] IndexError: list index out of range ```

Questions:

1. How does Cylinter know that it is a TMA run vs. a WSI run? does it need the params.yml file from MCMICRO?

[gjbaker]

Hi @josenimo ,

Thanks again for your comment. CyLinter looks for the dearray folder in deciding whether the MCMICRO input are TMA or WSI data; the params.yml file is not referred to in this process. From your config.yml file, it seems the sample names in the sampleMetadata field may need to be adjusted. In the context of MCMICRO output, these dictionary keys are meant to represent the file names in the quantification folder (without their file extensions). Try updating the filenames to see if that helps, and feel free to reach back out if anything else comes up.

[josenimo]

Hey @gjbaker ,

Thank you for clearing that up,

I have replaced the keys in the sampleMetadata dictionary in the config.yml. and now the error is another one, sorry for all the troubles

Stack Trace 2

``` (cylinter-env47) CMP06623:~ jnimoca$ cylinter /Volumes/ag_coscia/Jose/P21_Sonja_HN/P21E05_HN26/Cylinter_config.yml INFO: Reading configuration file INFO: Executing pipeline Running: INFO: ====================================================================== INFO: RUNNING MODULE: aggregateData INFO: Aborting; segmentation outlines for sample 11 not found. ```

Tree of my directory

``` (cylinter-env47) CMP06623:P21E05_HN26 jnimoca$ tree . ├── 20231213_Cylinter_Metadata.ipynb ├── Cylinter_config copy.txt ├── Cylinter_config.yml ├── Cylinter_output ├── Metadata_HN26.xlsx ├── background │   ├── P21E05_HN26_backsub.ome.tif │   └── markers_bs.csv ├── dearray │   ├── 1.ome.tif │   ├── 10.ome.tif │   ├── 11.ome.tif │   ├── 12.ome.tif │   ├── 13.ome.tif │   ├── 14.ome.tif │   ├── 15.ome.tif │   ├── 16.ome.tif │   ├── 17.ome.tif │   ├── 18.ome.tif │   ├── 19.ome.tif │   ├── 2.ome.tif │   ├── 20.ome.tif │   ├── 21.ome.tif │   ├── 22.ome.tif │   ├── 23.ome.tif │   ├── 24.ome.tif │   ├── 25.ome.tif │   ├── 26.ome.tif │   ├── 27.ome.tif │   ├── 28.ome.tif │   ├── 29.ome.tif │   ├── 3.ome.tif │   ├── 30.ome.tif │   ├── 31.ome.tif │   ├── 32.ome.tif │   ├── 33.ome.tif │   ├── 34.ome.tif │   ├── 35.ome.tif │   ├── 4.ome.tif │   ├── 5.ome.tif │   ├── 6.ome.tif │   ├── 7.ome.tif │   ├── 8.ome.tif │   ├── 9.ome.tif │   └── masks │   ├── 10_mask.tif │   ├── 11_mask.tif │   ├── 12_mask.tif │   ├── 13_mask.tif │   ├── 14_mask.tif │   ├── 15_mask.tif │   ├── 16_mask.tif │   ├── 17_mask.tif │   ├── 18_mask.tif │   ├── 19_mask.tif │   ├── 1_mask.tif │   ├── 20_mask.tif │   ├── 21_mask.tif │   ├── 22_mask.tif │   ├── 23_mask.tif │   ├── 24_mask.tif │   ├── 25_mask.tif │   ├── 26_mask.tif │   ├── 27_mask.tif │   ├── 28_mask.tif │   ├── 29_mask.tif │   ├── 2_mask.tif │   ├── 30_mask.tif │   ├── 31_mask.tif │   ├── 32_mask.tif │   ├── 33_mask.tif │   ├── 34_mask.tif │   ├── 35_mask.tif │   ├── 3_mask.tif │   ├── 4_mask.tif │   ├── 5_mask.tif │   ├── 6_mask.tif │   ├── 7_mask.tif │   ├── 8_mask.tif │   └── 9_mask.tif ├── illumination │   ├── Cycle_0.czi-dfp.tif │   ├── Cycle_0.czi-ffp.tif │   ├── Cycle_1.czi-dfp.tif │   ├── Cycle_1.czi-ffp.tif │   ├── Cycle_2.czi-dfp.tif │   ├── Cycle_2.czi-ffp.tif │   ├── Cycle_3.czi-dfp.tif │   ├── Cycle_3.czi-ffp.tif │   ├── Cycle_4.czi-dfp.tif │   ├── Cycle_4.czi-ffp.tif │   ├── Cycle_5.czi-dfp.tif │   └── Cycle_5.czi-ffp.tif ├── markers.csv ├── params │   ├── P21E05_params.yml │   └── P21E05_params_2.yml ├── qc │   ├── coreograph │   │   ├── TMA_MAP.tif │   │   └── centroidsY-X.txt │   ├── mesmer │   ├── metadata.yml │   ├── params.yml │   └── provenance │   ├── ashlar.log │   ├── ashlar.sh │   ├── backsub-1.log │   ├── backsub-1.sh │   ├── coreograph.log │   ├── coreograph.sh │   ├── mcquant-1.log │   ├── mcquant-1.sh │   ├── mcquant-10.log │   ├── mcquant-10.sh │   ├── mcquant-11.log │   ├── mcquant-11.sh │   ├── mcquant-12.log │   ├── mcquant-12.sh │   ├── mcquant-13.log │   ├── mcquant-13.sh │   ├── mcquant-14.log │   ├── mcquant-14.sh │   ├── mcquant-15.log │   ├── mcquant-15.sh │   ├── mcquant-16.log │   ├── mcquant-16.sh │   ├── mcquant-17.log │   ├── mcquant-17.sh │   ├── mcquant-18.log │   ├── mcquant-18.sh │   ├── mcquant-19.log │   ├── mcquant-19.sh │   ├── mcquant-2.log │   ├── mcquant-2.sh │   ├── mcquant-20.log │   ├── mcquant-20.sh │   ├── mcquant-21.log │   ├── mcquant-21.sh │   ├── mcquant-22.log │   ├── mcquant-22.sh │   ├── mcquant-23.log │   ├── mcquant-23.sh │   ├── mcquant-24.log │   ├── mcquant-24.sh │   ├── mcquant-25.log │   ├── mcquant-25.sh │   ├── mcquant-26.log │   ├── mcquant-26.sh │   ├── mcquant-27.log │   ├── mcquant-27.sh │   ├── mcquant-28.log │   ├── mcquant-28.sh │   ├── mcquant-29.log │   ├── mcquant-29.sh │   ├── mcquant-3.log │   ├── mcquant-3.sh │   ├── mcquant-30.log │   ├── mcquant-30.sh │   ├── mcquant-31.log │   ├── mcquant-31.sh │   ├── mcquant-32.log │   ├── mcquant-32.sh │   ├── mcquant-33.log │   ├── mcquant-33.sh │   ├── mcquant-34.log │   ├── mcquant-34.sh │   ├── mcquant-35.log │   ├── mcquant-35.sh │   ├── mcquant-4.log │   ├── mcquant-4.sh │   ├── mcquant-5.log │   ├── mcquant-5.sh │   ├── mcquant-6.log │   ├── mcquant-6.sh │   ├── mcquant-7.log │   ├── mcquant-7.sh │   ├── mcquant-8.log │   ├── mcquant-8.sh │   ├── mcquant-9.log │   ├── mcquant-9.sh │   ├── mesmer-1.log │   ├── mesmer-1.sh │   ├── mesmer-10.log │   ├── mesmer-10.sh │   ├── mesmer-11.log │   ├── mesmer-11.sh │   ├── mesmer-12.log │   ├── mesmer-12.sh │   ├── mesmer-13.log │   ├── mesmer-13.sh │   ├── mesmer-14.log │   ├── mesmer-14.sh │   ├── mesmer-15.log │   ├── mesmer-15.sh │   ├── mesmer-16.log │   ├── mesmer-16.sh │   ├── mesmer-17.log │   ├── mesmer-17.sh │   ├── mesmer-18.log │   ├── mesmer-18.sh │   ├── mesmer-19.log │   ├── mesmer-19.sh │   ├── mesmer-2.log │   ├── mesmer-2.sh │   ├── mesmer-20.log │   ├── mesmer-20.sh │   ├── mesmer-21.log │   ├── mesmer-21.sh │   ├── mesmer-22.log │   ├── mesmer-22.sh │   ├── mesmer-23.log │   ├── mesmer-23.sh │   ├── mesmer-24.log │   ├── mesmer-24.sh │   ├── mesmer-25.log │   ├── mesmer-25.sh │   ├── mesmer-26.log │   ├── mesmer-26.sh │   ├── mesmer-27.log │   ├── mesmer-27.sh │   ├── mesmer-28.log │   ├── mesmer-28.sh │   ├── mesmer-29.log │   ├── mesmer-29.sh │   ├── mesmer-3.log │   ├── mesmer-3.sh │   ├── mesmer-30.log │   ├── mesmer-30.sh │   ├── mesmer-31.log │   ├── mesmer-31.sh │   ├── mesmer-32.log │   ├── mesmer-32.sh │   ├── mesmer-33.log │   ├── mesmer-33.sh │   ├── mesmer-34.log │   ├── mesmer-34.sh │   ├── mesmer-35.log │   ├── mesmer-35.sh │   ├── mesmer-4.log │   ├── mesmer-4.sh │   ├── mesmer-5.log │   ├── mesmer-5.sh │   ├── mesmer-6.log │   ├── mesmer-6.sh │   ├── mesmer-7.log │   ├── mesmer-7.sh │   ├── mesmer-8.log │   ├── mesmer-8.sh │   ├── mesmer-9.log │   └── mesmer-9.sh ├── quantification │   ├── 1--mesmer_cell.csv │   ├── 10--mesmer_cell.csv │   ├── 11--mesmer_cell.csv │   ├── 12--mesmer_cell.csv │   ├── 13--mesmer_cell.csv │   ├── 14--mesmer_cell.csv │   ├── 15--mesmer_cell.csv │   ├── 16--mesmer_cell.csv │   ├── 17--mesmer_cell.csv │   ├── 18--mesmer_cell.csv │   ├── 19--mesmer_cell.csv │   ├── 2--mesmer_cell.csv │   ├── 20--mesmer_cell.csv │   ├── 21--mesmer_cell.csv │   ├── 22--mesmer_cell.csv │   ├── 23--mesmer_cell.csv │   ├── 24--mesmer_cell.csv │   ├── 25--mesmer_cell.csv │   ├── 26--mesmer_cell.csv │   ├── 27--mesmer_cell.csv │   ├── 28--mesmer_cell.csv │   ├── 29--mesmer_cell.csv │   ├── 3--mesmer_cell.csv │   ├── 30--mesmer_cell.csv │   ├── 31--mesmer_cell.csv │   ├── 32--mesmer_cell.csv │   ├── 33--mesmer_cell.csv │   ├── 34--mesmer_cell.csv │   ├── 35--mesmer_cell.csv │   ├── 4--mesmer_cell.csv │   ├── 5--mesmer_cell.csv │   ├── 6--mesmer_cell.csv │   ├── 7--mesmer_cell.csv │   ├── 8--mesmer_cell.csv │   └── 9--mesmer_cell.csv ├── raw │   ├── Cycle_0.czi │   ├── Cycle_1.czi │   ├── Cycle_2.czi │   ├── Cycle_3.czi │   ├── Cycle_4.czi │   └── Cycle_5.czi ├── registration │   └── P21E05_HN26.ome.tif ├── scripts │   └── P21E04_1.sh └── segmentation ├── mesmer-1 │   └── cell.tif ├── mesmer-10 │   └── cell.tif ├── mesmer-11 │   └── cell.tif ├── mesmer-12 │   └── cell.tif ├── mesmer-13 │   └── cell.tif ├── mesmer-14 │   └── cell.tif ├── mesmer-15 │   └── cell.tif ├── mesmer-16 │   └── cell.tif ├── mesmer-17 │   └── cell.tif ├── mesmer-18 │   └── cell.tif ├── mesmer-19 │   └── cell.tif ├── mesmer-2 │   └── cell.tif ├── mesmer-20 │   └── cell.tif ├── mesmer-21 │   └── cell.tif ├── mesmer-22 │   └── cell.tif ├── mesmer-23 │   └── cell.tif ├── mesmer-24 │   └── cell.tif ├── mesmer-25 │   └── cell.tif ├── mesmer-26 │   └── cell.tif ├── mesmer-27 │   └── cell.tif ├── mesmer-28 │   └── cell.tif ├── mesmer-29 │   └── cell.tif ├── mesmer-3 │   └── cell.tif ├── mesmer-30 │   └── cell.tif ├── mesmer-31 │   └── cell.tif ├── mesmer-32 │   └── cell.tif ├── mesmer-33 │   └── cell.tif ├── mesmer-34 │   └── cell.tif ├── mesmer-35 │   └── cell.tif ├── mesmer-4 │   └── cell.tif ├── mesmer-5 │   └── cell.tif ├── mesmer-6 │   └── cell.tif ├── mesmer-7 │   └── cell.tif ├── mesmer-8 │   └── cell.tif └── mesmer-9 └── cell.tif 113 directories, 496 files

Hopefully this helps to understand why is it not working with me.

[gjbaker]

There should be a subfolder in the qc directory called s3seg. This is where the segmentation outlines are stored that CyLinter superimposes on images so users can vet segmentation quality. The program is throwing the error, because that folder seems to be absent. It would be possible to run the program without segmentation outlines, but would take some commenting out of lines involving their reading by the program.

[josenimo]

ah I understand, right now I am using mostly mesmer or cellpose with MCMICRO. I also think that the segmentation masks are critical for quality control checking. I could try changing the code so that the parsing of the segmentation masks can be adapted to the output of mesmer segmentation. I guess this is why I wondered if Cylinter read the params.yml.