d33bs
commented
11 months ago Hi @sbamin, thank you so much for opening this issue and for sharing along the details of your efforts here! I took some time to look into this and wanted to share my findings so far. I'm currently running into challenges which are specific to my limited local hardware resources and chunk sizes which may not apply to your execution environment (you may see different results based on what I share).
I created a gist for some troubleshooting I performed which may be found here: https://gist.github.com/d33bs/5dd2e5384cee6ea5907e4cd94ae51ff1 . I focused only on the "orig" dataset, presuming this might be preferred for you (not needing additional steps to preprocess the data before it reaches CytoTable). Please note: I used Docker for container-based troubleshooting as I'm more familiar with it than Apptainer and to balance providing feedback here sooner.
Answering a question you raised which may provide more context on the above:
I believe the way collate feature works for cytominer and cytotable is to look for matching string in file name...
- CytoTable gathers and groups input data by filename for CSV's and filename + tablename for SQLite.
- Each filename is filtered by the
compartmentsandmetadataparameters which are lists of strings provided to CytoTable in order to understand what will be processed for inclusion (cytotable.convert(compartments : List[str] = [...], metadata : List[str] = [...])). Filenames matching those inputs are grouped together for later data concatenation and/or join work.- Compartments and image metadata have relationships which are important to the resulting datasets created. See here for more description.
- Currently, including "compartment-type" files for processing with CytoTable (i.e.
cytoplasm_bf,cytoplasm_dying,cytoplasm_live, etc.) will I believe require they be treated like "compartment groups" or distinct sets of data. I tried but was unable to perform a SQL UNION on like-files of the same compartment type due to I believe the column-naming scheme found in each.
See below for a snippet of what might work for you in this circumstance. Please note: this is based on many assumptions, and the result may not be what you desire. Don't hesitate to let me know if you have questions or would like to see something different happen within CytoTable.
From cytotable_test.py:
"""
Perform a test using cytotable as part of:
https://github.com/cytomining/CytoTable/issues/111
"""
import cytotable
cytotable.convert(
source_path="toyset/data/orig",
source_datatype="csv",
dest_path="take1.parquet",
dest_datatype="parquet",
preset="cellprofiler_csv",
# note, here we override some of the above preset options
metadata=["hcssyn_seg_image"],
compartments=[
"hcssyn_seg_cyto_bf",
"hcssyn_seg_cyto_dying",
"hcssyn_seg_cyto_live",
"hcssyn_seg_cells_bf",
"hcssyn_seg_cells_dying",
"hcssyn_seg_cells_live",
"hcssyn_seg_nuclei_bf",
"hcssyn_seg_nuclei_dead",
"hcssyn_seg_nuclei_live",
],
chunk_columns=["Metadata_Plate"],
joins="""
WITH Image_filtered AS (
SELECT
Metadata_Hcssyn_seg_image_ImageNumber,
Metadata_Well,
Metadata_Plate
FROM
read_parquet('hcssyn_seg_image.parquet')
)
SELECT
*
FROM
Image_filtered AS Hcssyn_seg_image
/* compartment type: bf */
LEFT JOIN read_parquet('hcssyn_seg_cyto_bf.parquet') as Hcssyn_seg_cyto_bf ON
Hcssyn_seg_cyto_bf.Metadata_Hcssyn_seg_cyto_bf_ImageNumber = Hcssyn_seg_image.Metadata_Hcssyn_seg_image_ImageNumber
LEFT JOIN read_parquet('hcssyn_seg_cells_bf.parquet') as Hcssyn_seg_cells_bf ON
Hcssyn_seg_cells_bf.Metadata_Hcssyn_seg_cells_bf_ImageNumber = Hcssyn_seg_cyto_bf.Metadata_Hcssyn_seg_cyto_bf_ImageNumber
AND Hcssyn_seg_cells_bf.Hcssyn_seg_cells_bf_Number_Object_Number = Hcssyn_seg_cyto_bf.Hcssyn_seg_cyto_bf_Parent_cells_bf
LEFT JOIN read_parquet('hcssyn_seg_nuclei_bf.parquet') as Hcssyn_seg_nuclei_bf ON
Hcssyn_seg_nuclei_bf.Metadata_Hcssyn_seg_nuclei_bf_ImageNumber = Hcssyn_seg_cyto_bf.Metadata_Hcssyn_seg_cyto_bf_ImageNumber
AND Hcssyn_seg_nuclei_bf.Hcssyn_seg_nuclei_bf_Number_Object_Number = Hcssyn_seg_cyto_bf.Hcssyn_seg_cyto_bf_Parent_nuclei_bf
/* compartment type: dying or dead */
LEFT JOIN read_parquet('hcssyn_seg_cyto_dying.parquet') as Hcssyn_seg_cyto_dying ON
Hcssyn_seg_cyto_dying.Metadata_Hcssyn_seg_cyto_dying_ImageNumber = Hcssyn_seg_image.Metadata_Hcssyn_seg_image_ImageNumber
LEFT JOIN read_parquet('hcssyn_seg_cells_dying.parquet') as Hcssyn_seg_cells_dying ON
Hcssyn_seg_cells_dying.Metadata_Hcssyn_seg_cells_dying_ImageNumber = Hcssyn_seg_cyto_dying.Metadata_Hcssyn_seg_cyto_dying_ImageNumber
AND Hcssyn_seg_cells_dying.Hcssyn_seg_cells_dying_Number_Object_Number = Hcssyn_seg_cyto_dying.Hcssyn_seg_cyto_dying_Parent_cells_dying
LEFT JOIN read_parquet('hcssyn_seg_nuclei_dead.parquet') as Hcssyn_seg_nuclei_dead ON
Hcssyn_seg_nuclei_dead.Metadata_Hcssyn_seg_nuclei_dead_ImageNumber = Hcssyn_seg_cyto_dying.Metadata_Hcssyn_seg_cyto_dying_ImageNumber
AND Hcssyn_seg_nuclei_dead.Hcssyn_seg_nuclei_dead_Number_Object_Number = Hcssyn_seg_cyto_dying.Hcssyn_seg_cyto_dying_Parent_nuclei_dead
/* compartment type: live */
LEFT JOIN read_parquet('hcssyn_seg_cyto_live.parquet') as Hcssyn_seg_cyto_live ON
Hcssyn_seg_cyto_live.Metadata_Hcssyn_seg_cyto_live_ImageNumber = Hcssyn_seg_image.Metadata_Hcssyn_seg_image_ImageNumber
LEFT JOIN read_parquet('hcssyn_seg_cells_live.parquet') as Hcssyn_seg_cells_live ON
Hcssyn_seg_cells_live.Metadata_Hcssyn_seg_cells_live_ImageNumber = Hcssyn_seg_cyto_live.Metadata_Hcssyn_seg_cyto_live_ImageNumber
AND Hcssyn_seg_cells_live.Hcssyn_seg_cells_live_Number_Object_Number = Hcssyn_seg_cyto_live.Hcssyn_seg_cyto_live_Parent_cells_live
LEFT JOIN read_parquet('hcssyn_seg_nuclei_live.parquet') as Hcssyn_seg_nuclei_live ON
Hcssyn_seg_nuclei_live.Metadata_Hcssyn_seg_nuclei_live_ImageNumber = Hcssyn_seg_cyto_live.Metadata_Hcssyn_seg_cyto_live_ImageNumber
AND Hcssyn_seg_nuclei_live.Hcssyn_seg_nuclei_live_Number_Object_Number = Hcssyn_seg_cyto_live.Hcssyn_seg_cyto_live_Parent_nuclei_live
""",
)In case it's helpful, I recommend also looking at the results of using CytoTable with convert(..., join=False, ...). This will provide a multi-file result that may provide a different perspective on analysis or implementation possibilities.
sbamin
I am trying to collate csv files from a phenotypic screen into a single parquet or sqlite file using cytotable or cytominer (based on your suggestion). Some details on our study:
I have cellprofiler output in csv format with each folder containing well-level run of cellprofiler across multiple 384-well plates. Each well-level folder contains csv data for segmented nuclei, cells, and cytoplasm using brightfield, NucBlue, and NucGreen with respective file names ending
bf.csv,live.csv, and(dead|dying).csv.I have collated well-level summary data from
Image.csv. While that works to infer aggregate effects per knockout gene, I like to leverage cell or segment-level data from respective NucBlue (live) and NucGreen (dying or dead) stains, especially for nuclei and less so for cells and cytoplasm (named cyto).While running cellprofiler, I made a mistake for one of output csv and named it nuclei_dead.csv instead of nuclei_dying.csv. I believe the way collate feature works for cytominer and cytotable is to look for matching string in file name (? and column names for those files), and then start merging data at well and then plate level (depending on folder structure).
For now, I get the following error regardless of if I used original data (with inconsistent filenames) or after renaming _hcssyn_seg_nucleidying.csv to _hcssyn_seg_nucleidead.csv (but not replacing dying with dead within column names!). I am not sure if error is related to not replacing dying to dead within column names or something else but would be much help if there is a way to overcome this issue.
I have shared toyset data via email.
gist: apptainer definition file to build cytotable_0.0.1p2.sif, using commit: e74a6785ece152008d91e5f4bd59b8f5fc8b9314
I am able to successfully run tutorial code with above apptainer image.