search

dcjones / proseg

Probabilistic cell segmentation for in situ spatial transcriptomics
Other
45 stars 2 forks source link

xenium data issue #2

Closed williamsdrake closed 6 months ago

williamsdrake commented 6 months ago

Hi, thanks for the nice package. I'm trying to use proseg with some xenium datasets following the instructions on your page. I've generated the proseg output using: proseg /data/xenium/outs/transcripts.csv.gz --xenium Then I converted the proseg output to baysor compatible formatting like so: proseg-to-baysor transcript-metadata.csv.gz cell-polygons.geojson.gz --output-transcript-metadata baysor-transcript-metadata.csv --output-cell-polygons baysor-cell-polygons.geojson

Finally, I tried to generate a new xenium bundle with: xeniumranger import-segmentation --id s1r1_proseg --xenium-bundle /data/williamsdrw/xenium/run1_5um/s1r1/outs --viz-polygons baysor-cell-polygons.geojson --transcript-assignment baysor-transcript-metadata.csv --units microns

However, I get an error when doing this: Log message: Traceback (most recent call last): File "/usr/local/apps/xeniumranger/2.0.0.12/external/martian/adapters/python/martian_shell.py", line 648, in _main stage.main() File "/usr/local/apps/xeniumranger/2.0.0.12/external/martian/adapters/python/martian_shell.py", line 617, in main self._run( File "/usr/local/apps/xeniumranger/2.0.0.12/external/martian/adapters/python/martian_shell.py", line 564, in _run cmd() File "/usr/local/apps/xeniumranger/2.0.0.12/external/martian/adapters/python/martian_shell.py", line 618, in lambda: self._module.join(args, outs, chunk_defs, chunk_outs) File "/usr/local/apps/xeniumranger/2.0.0.12/mro/insitu/stages/xr_import_transcripts/init.py", line 97, in join import_transcript_assignments.join( File "/usr/local/apps/xeniumranger/2.0.0.12/lib/python/turing/engine/xenium_ranger/stages/import_transcript_assignments.py", line 69, in join results = import_baysor_transcript_assignments( File "/usr/local/apps/xeniumranger/2.0.0.12/lib/python/turing/engine/xenium_ranger/import_segmentation/import_transcripts.py", line 83, in import_baysor_transcript_assignments baysor_results = BaysorSegmentationResults.deserialize( File "/usr/local/apps/xeniumranger/2.0.0.12/lib/python/turing/engine/xenium_ranger/import_segmentation/baysor.py", line 396, in deserialize segmentation = BaysorSegmentation.deserialize(segmentation_path) File "/usr/local/apps/xeniumranger/2.0.0.12/lib/python/turing/engine/xenium_ranger/import_segmentation/baysor.py", line 129, in deserialize .collect() File "/usr/local/apps/xeniumranger/2.0.0.12/external/anaconda/lib/python3.10/site-packages/polars/utils/deprecation.py", line 100, in wrapper return function(*args, **kwargs) File "/usr/local/apps/xeniumranger/2.0.0.12/external/anaconda/lib/python3.10/site-packages/polars/lazyframe/frame.py", line 1788, in collect return wrap_df(ldf.collect()) polars.exceptions.SchemaError: invalid series dtype: expected Utf8, got i64

Although the error is not with your package, I'm wondering if something happened during the generation of the proseg files that would lead to this error. Any ideas? Thanks in advance

dcjones commented 6 months ago

Thanks for the report! The proseg-to-bayser command was slightly out of step with the latest xenium ranger version. I just released version 1.0.1 that fixes this issue, you can run cargo install proseg to update.

No need to re-run proseg again, just re-run proseg-to-baysor and the rest and it ought to work.

saito-k1 commented 6 months ago

Hello! I had the same issue as the person above. I just reinstalled proseg and I am attempting to rerun proseg-to-baysor. I get a new error stating:

"thread 'main' panicked at /Users/Kai/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow2-0.18.0/src/array/physical_binary.rs:208:41: called Result::unwrap() on an Err value: Overflow note: run with RUST_BACKTRACE=1 environment variable to display a backtrace"

when I run:

proseg-to-baysor \ transcript-metadata.csv.gz \ cell-polygons.geojson.gz \ --output-transcript-metadata baysor-transcript-metadata.csv \ --output-cell-polygons baysor-cell-polygons.geojson

dcjones commented 6 months ago

Hmm that's odd, can you run the same command prepended with RUST_BACKTRACE=1, (like RUST_BACKTRACE=1 proseg-to-baysor ...), and paste the output here?

saito-k1 commented 6 months ago

Thank you so much for the quick response! When I run it prepended with RUST_BACKTRACE=1, I get the output: "thread 'main' panicked at /Users/Kai/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow2-0.18.0/src/array/physical_binary.rs:208:41: called Result::unwrap() on an Err value: Overflow stack backtrace: 0: _rust_begin_unwind 1: core::panicking::panic_fmt 2: core::result::unwrap_failed 3: <arrow2::array::utf8::mutable_values::MutableUtf8ValuesArray as core::iter::traits::collect::FromIterator

>::from_iter 4: proseg_to_baysor::main note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace."

I then reran it with RUST_BACKTRACE=full and receive the output: "thread 'main' panicked at /Users/Kai/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow2-0.18.0/src/array/physical_binary.rs:208:41: called Result::unwrap() on an Err value: Overflow stack backtrace: 0: 0x1024e737c - ::fmt::h0aa20ca08aeb683c 1: 0x1024fe9a4 - core::fmt::write::h168dbafcf35bac68 2: 0x1024e574c - std::io::Write::write_fmt::hdb0dd3f09dcf2281 3: 0x1024e71b0 - std::sys_common::backtrace::print::h57b289e4b951ee17 4: 0x1024e85f4 - std::panicking::default_hook::{{closure}}::h783b6c512154ec65 5: 0x1024e833c - std::panicking::default_hook::hcdfa9e1e0f234a4f 6: 0x1024e8a30 - std::panicking::rust_panic_with_hook::h9aea678ca49d64cf 7: 0x1024e8924 - std::panicking::begin_panic_handler::{{closure}}::ha16c3377e66deceb 8: 0x1024e7800 - std::sys_common::backtrace::__rust_end_short_backtrace::hea8fdda1ea8a4c0e 9: 0x1024e86ac - _rust_begin_unwind 10: 0x102508a44 - core::panicking::panic_fmt::h1cb43b60f5788132 11: 0x102508d88 - core::result::unwrap_failed::h71a35eff74d84b68 12: 0x102230808 - <arrow2::array::utf8::mutable_values::MutableUtf8ValuesArray as core::iter::traits::collect::FromIterator

>::from_iter::h951ef6dc5807e04f 13: 0x102223adc - proseg_to_baysor::main::hf1c2c208ba6c1004 14: 0x102244454 - std::sys_common::backtrace::__rust_begin_short_backtrace::h54af7924eb282f8e 15: 0x10223b758 - std::rt::lang_start::{{closure}}::hbf1d8b5aebf92de5 16: 0x1024e21e0 - std::rt::lang_start_internal::h0e09503d2b7f298e 17: 0x102226cdc - _main"

williamsdrake commented 6 months ago

For what it's worth, reinstalling proseg seems to have worked for me. The xeniumranger command hasn't finished running but it's progressed more than it did last time.

dcjones commented 6 months ago

Glad that seemed to work @williamsdrake

@saito-k1 Thanks for the backtrace. I think I know what the issue is and put up a new version to fix it. Give that a try and let me know if it worked.

saito-k1 commented 6 months ago

It's working now, thank you so much for your help!

dcjones commented 6 months ago

Great, thanks for trying out proseg!

krademaker commented 6 months ago

Hi,

Thanks for the solution posted here, I ran into exactly the same issue on my Xenium data and the updated proseg-to-baysor seems to have partially solved that issue. However, I encountered a different issue in the xeniumranger import-segmentation step, and I wanted to check if this is familiar or easily fixed?

After submitting:

xeniumranger import-segmentation \
    --id=output_dir_KR_XEN \
    --xenium-bundle=/path/to//output-XETG00055__0003505__Region_1__20230310__172140 \
    --viz-polygons=${WORK_DIR}/baysor-cell-polygons.geojson \
    --transcript-assignment=${WORK_DIR}/baysor-transcript-metadata.csv \
    --units=microns

XR fails on the ... step due to permission issues on an LSF file system, with the LSF log:

Martian Runtime - v4.0.12
Serving UI at http://node-13-11:36285?auth=6Q4mfHDserq-KB4Cin5bAMNTdqLJbYe8p71ErTkELBo

Running preflight checks (please wait)...
2024-05-09 10:22:51 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_SETUP_INPUTS
2024-05-09 10:22:51 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_SETUP_INPUTS.fork0.split
2024-05-09 10:23:00 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_SETUP_INPUTS
2024-05-09 10:23:00 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_SETUP_INPUTS.fork0.join
2024-05-09 10:23:47 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_SETUP_INPUTS
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
2024-05-09 10:23:47 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.split
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.SETUP_COUNTER_CS
2024-05-09 10:23:47 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.SETUP_COUNTER_CS.fork0.split
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT.SEGMENT_NUCLEI
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_MORPHOLOGY_PLOTS
2024-05-09 10:23:47 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_MORPHOLOGY_PLOTS.fork0.split
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.IMPORT_CELL_SEGMENTATION_DATASET
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL_TRANSCRIPTS
2024-05-09 10:23:47 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL_TRANSCRIPTS.fork0.split
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT.SETUP_CELL_SEGMENTOR_FINALIZER
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.IMPORT_NUCLEAR_SEGMENTATION_DATASET
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT._CELL_SEGMENTOR_FINALIZER.SEGMENT_CELLS
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT._CELL_SEGMENTOR_FINALIZER.SEGMENT_INTERIOR
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.PAIR_CELLS_NUCLEI
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT._CELL_SEGMENTOR_FINALIZER.PAIR_CELLS_NUCLEI
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.EXPAND_NUCLEI
2024-05-09 10:23:47 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.GENERATE_CELLS_DATASET
2024-05-09 10:23:48 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT._CELL_SEGMENTOR_FINALIZER.EXPAND_NUCLEI
2024-05-09 10:23:48 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RESEGMENT._CELL_SEGMENTOR_FINALIZER.GENERATE_CELLS_DATASET
2024-05-09 10:23:48 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.SIMPLE_MERGE_METRICS
2024-05-09 10:23:48 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_SEGMENTATION.XR_FINALIZE_IMPORT_SEGMENTATION
2024-05-09 10:24:22 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
2024-05-09 10:24:22 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.join
2024-05-09 10:24:26 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.SETUP_COUNTER_CS
2024-05-09 10:24:26 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.SETUP_COUNTER_CS.fork0.join
2024-05-09 10:24:34 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_MORPHOLOGY_PLOTS
2024-05-09 10:24:34 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_MORPHOLOGY_PLOTS.fork0.join
2024-05-09 10:24:38 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL_TRANSCRIPTS
2024-05-09 10:24:38 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL_TRANSCRIPTS.fork0.join
2024-05-09 10:25:20 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
2024-05-09 10:25:20 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
2024-05-09 10:25:20 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET.fork0.chnk0.main
2024-05-09 10:25:21 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
2024-05-09 10:25:21 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS.fork0.chnk0.main
2024-05-09 10:25:24 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.SETUP_COUNTER_CS
2024-05-09 10:25:33 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_MORPHOLOGY_PLOTS
2024-05-09 10:26:06 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL_TRANSCRIPTS
2024-05-09 10:26:07 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
2024-05-09 10:26:08 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_GENE_MATRIX
2024-05-09 10:26:08 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_GENE_MATRIX.fork0.split
2024-05-09 10:26:08 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.COMPUTE_NUCLEAR_DISTANCE
2024-05-09 10:26:08 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.COMPUTE_NUCLEAR_DISTANCE.fork0.split
2024-05-09 10:26:08 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.WRITE_CELL_NUCLEI_BOUNDARIES_FILES
2024-05-09 10:26:08 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.WRITE_CELL_NUCLEI_BOUNDARIES_FILES.fork0.split
2024-05-09 10:26:09 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
2024-05-09 10:26:14 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_GENE_MATRIX
2024-05-09 10:26:14 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_GENE_MATRIX.fork0.join
2024-05-09 10:26:18 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.COMPUTE_NUCLEAR_DISTANCE
2024-05-09 10:26:18 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.COMPUTE_NUCLEAR_DISTANCE.fork0.join
2024-05-09 10:26:26 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.WRITE_CELL_NUCLEI_BOUNDARIES_FILES
2024-05-09 10:26:26 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.WRITE_CELL_NUCLEI_BOUNDARIES_FILES.fork0.join
2024-05-09 10:26:32 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_GENE_MATRIX
2024-05-09 10:26:32 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CONCATENATE_MATRIX
2024-05-09 10:26:32 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CONCATENATE_MATRIX.fork0.chnk0.main
2024-05-09 10:26:39 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.COMPUTE_NUCLEAR_DISTANCE
2024-05-09 10:26:39 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_TRANSCRIPTS_CSV
2024-05-09 10:26:39 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_TRANSCRIPTS_CSV.fork0.split
2024-05-09 10:26:50 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.WRITE_CELL_NUCLEI_BOUNDARIES_FILES
2024-05-09 10:26:53 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CONCATENATE_MATRIX
2024-05-09 10:26:53 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_DECODING_METRICS_CS
2024-05-09 10:26:53 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_DECODING_METRICS_CS.fork0.split
2024-05-09 10:26:53 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_MTX
2024-05-09 10:26:53 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_MTX.fork0.split
2024-05-09 10:26:53 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_ZARR
2024-05-09 10:26:53 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_ZARR.fork0.split
2024-05-09 10:26:53 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.SETUP_ANALYZER
2024-05-09 10:26:53 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.SETUP_ANALYZER.fork0.chnk0.main
2024-05-09 10:26:58 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_TRANSCRIPTS_CSV
2024-05-09 10:26:58 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_TRANSCRIPTS_CSV.fork0.join
2024-05-09 10:27:14 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_DECODING_METRICS_CS
2024-05-09 10:27:14 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_DECODING_METRICS_CS.fork0.join
2024-05-09 10:27:20 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_MTX
2024-05-09 10:27:20 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_MTX.fork0.join
2024-05-09 10:27:26 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_ZARR
2024-05-09 10:27:26 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_ZARR.fork0.join
2024-05-09 10:27:29 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.SETUP_ANALYZER
2024-05-09 10:27:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA
2024-05-09 10:27:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA.fork0.split
2024-05-09 10:27:40 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_TRANSCRIPTS_CSV
2024-05-09 10:28:02 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_DECODING_METRICS_CS
2024-05-09 10:28:03 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_SAMPLE_METRICS
2024-05-09 10:28:03 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_SAMPLE_METRICS.fork0.split
2024-05-09 10:28:11 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_MTX
2024-05-09 10:28:20 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.CREATE_MATRIX_ZARR
2024-05-09 10:28:21 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA
2024-05-09 10:28:21 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA.fork0.chnk0.main
2024-05-09 10:28:31 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_SAMPLE_METRICS
2024-05-09 10:28:31 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_SAMPLE_METRICS.fork0.join
2024-05-09 10:28:31 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA
2024-05-09 10:28:32 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA.fork0.join
2024-05-09 10:28:43 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.METRIC_CALCULATOR_CS.GENERATE_SAMPLE_METRICS
2024-05-09 10:28:43 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_PCA
2024-05-09 10:28:43 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_UMAP
2024-05-09 10:28:43 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_UMAP.fork0.split
2024-05-09 10:28:43 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_GRAPH_CLUSTERING
2024-05-09 10:28:43 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.split
2024-05-09 10:28:43 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_TSNE
2024-05-09 10:28:44 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS
2024-05-09 10:28:44 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.split
2024-05-09 10:28:44 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_UMAP
2024-05-09 10:28:44 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_UMAP.fork0.chnk0.main
2024-05-09 10:28:44 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_GRAPH_CLUSTERING
2024-05-09 10:28:44 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.chnk0.main
2024-05-09 10:28:50 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk0.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk1.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk2.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk3.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk4.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk5.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk6.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk7.main
2024-05-09 10:28:50 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS.fork0.chnk8.main
2024-05-09 10:29:28 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_UMAP
2024-05-09 10:29:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_UMAP.fork0.join
2024-05-09 10:29:45 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_GRAPH_CLUSTERING
2024-05-09 10:29:45 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.join
2024-05-09 10:29:50 [runtime] (failed)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.INSITU_ANALYZER.RUN_KMEANS

[error] Pipestance failed. Error log at:
output_dir_KR_XEN/XENIUM_RANGER_CS/INSITU_COUNTER_CS/INSITU_COUNTER_CORE/INSITU_ANALYZER/RUN_KMEANS/fork0/chnk0-ua6d93c9752/_errors

Log message:
Traceback (most recent call last):
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/martian/adapters/python/martian_shell.py", line 648, in _main
    stage.main()
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/martian/adapters/python/martian_shell.py", line 607, in main
    self._run(lambda: self._module.main(args, outs))
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/martian/adapters/python/martian_shell.py", line 564, in _run
    cmd()
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/martian/adapters/python/martian_shell.py", line 607, in <lambda>
    self._run(lambda: self._module.main(args, outs))
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/cellranger/mro/rna/stages/analyzer/run_kmeans/__init__.py", line 115, in main
    with analysis_io.open_h5_for_writing(outs.kmeans_h5) as f:
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/cellranger/lib/python/cellranger/analysis/io.py", line 192, in open_h5_for_writing
    return tables.open_file(filename, "w", filters=filters)
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/anaconda/lib/python3.10/site-packages/tables/file.py", line 300, in open_file
    return File(filename, mode, title, root_uep, filters, **kwargs)
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/cellranger/lib/python/cellranger/wrapped_tables.py", line 54, in wrapper
    return function(*args, **kwargs)
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/anaconda/lib/python3.10/site-packages/tables/file.py", line 750, in __init__
    self._g_new(filename, mode, **params)
  File "tables/hdf5extension.pyx", line 366, in tables.hdf5extension.File._g_new
  File "/software/cellgen/team283/kr23/xeniumranger-xenium2.0/external/anaconda/lib/python3.10/site-packages/tables/utils.py", line 161, in check_file_access
    raise PermissionError(
PermissionError: directory ``/nfs/team283/kr23/github/safari_ASD/segmentation/proseg/KR_XEN_FFPE_Hs_cell_pellet_SAT/output_dir_KR_XEN/XENIUM_RANGER_CS/INSITU_COUNTER_CS/INSITU_COUNTER_CORE/INSITU_ANALYZER/RUN_KMEANS/fork0/chnk0-ua6d93c9752/files`` exists but it can not be written

Is this a known issue or something people have found a fix for? It seems like the permission error concerns a temporary output folder (output_dir_KR_XEN/XENIUM_RANGER_CS/INSITU_COUNTER_CS/INSITU_COUNTER_CORE/INSITU_ANALYZER/RUN_KMEANS/fork0/chnk0-ua6d93c9752/_errors) , so I don't know whether changing folder permissions can account for that?

Thank you very much, Koen

dcjones commented 6 months ago

Hi @krademaker

I haven't seen this specific error before. I don't think it's specific to proseg, so I'd suggest asking 10X if they have any ideas.

krademaker commented 6 months ago

Hello @dcjones, thanks for that. I figured out it was a stand-alone issue that has been fixed now. It had indeed nothing to do with Proseg itself. Proseg itself runs well and is very user-friendly!