Open shntnu opened 2 years ago
shntnu
commented
2 years ago We are having this discussion in Slack https://broadinstitute.slack.com/archives/C3QFQ3WQM/p1663329404078119?thread_ts=1663317811.106069&cid=C3QFQ3WQM
@EchteRobert will report back here with our conclusion here, once we are set
EchteRobert
commented
2 years ago You can find the main conclusion of this issue below.
Beth (she/her)
I don't think it's a known issue but based on the error messages I see how it happened. Was likely a pilot batch and/or a batch that was rerun later would be my guess, and at the time the 1) MeasureImageQuality modules were turned off (but not deleted) and 2) a couple of the measurement modules were rearranged) and 3) DNA was un-checked in the MeasureObjectIntensityDistribution module
Supporting discussion (for the full discussion see the Slack thread)
shantanu
The difference is 1781-1745=36 features, all of which are RadialDistribution features in the DNA channel across all the three compartments. I assume this is in the SQLite files, and not in the aggregated CSV files? Can you paste the list of plates where these features are missing?
Robert
Yes, this is in the SQLite files. I have not checked the aggregated CSV files. They are missing in these plates: SQ00015116, SQ00015117, SQ00015118, SQ00015119 (and I expect also SQ00015125 due to it having the same platemap but I haven’t downloaded this one yet)
Beth
https://broadinstitute.slack.com/archives/C3QFDHXC4/p1466877949000013 Looks like those were part of the very first batch of plates So my guess is that the pipeline got sightly changed after
shantanu
I confirm that the aggregated files have the same issue
x <- read_csv("https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/workspace/backend/2016_04_01_a549_48hr_batch1/SQ00015116/SQ00015116.csv", n_max = 2) Rows: 2 Columns: 1749 y <- read_csv("https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/workspace/backend/2016_04_01_a549_48hr_batch1/SQ00015100/SQ00015100.csv", n_max = 2) Rows: 2 Columns: 1785 > 1747-1785 [1] -36
Beth
Yeah, the modules in question were likely not run, which is why they don't have module execution times or error message reports in the image features
shantanu
Ah that's right For the record, these are the modules: {'Image_ExecutionTime_03MeasureImageQuality', 'Image_ExecutionTime_04MeasureImageQuality', 'Image_ExecutionTime_11MeasureObjectIntensityDistribution', 'Image_ExecutionTime_12MeasureObjectIntensity', 'Image_ExecutionTime_15MeasureObjectNeighbors',
shantanu
But out of the last 3, I looks like only Image_ExecutionTime_11MeasureObjectIntensityDistributionwas the one that wasn't run, right Beth (she/her)? Because all are RadialDistribution features in the DNA channel across all the three compartments, which is what we measure in MeasureObjectIntensityDistribution (edited)
shantanu
But there are no missing features wrt MeasureObjectIntensity and MeasureObjectNeighbors
Beth (she/her)
Right, but if the order if those was changed, you wouldn't have an exact match column (edited) But you might have a similarly named one with just a slightly different number Number=position in the pipeline Probably intensity distribution WAS run, just with DNA not checked My guess Robert is that the other plates mentioned in the thing I linked will have the same issue
shantanu
aws s3 ls s3://cellpainting-gallery/cpg0004-lincs/broad/workspace/backend/2016_04_01_a549_48hr_batch1/%7Ctr -s " "|cut -c6-15|sort > /tmp/platelist
parallel -a /tmp/platelist "echo -n {1}; aws s3 cp s3://cellpainting-gallery/cpg0004-lincs/broad/workspace/backend/2016_04_01_a549_48hr_batch1/{1}/{1}.csv -|csvcut -n|wc -l"|grep -v "download failed"|tr -s " "|tr " " ","|csvcut -c 2,1|sort -n > /tmp/countscat /t
mp/counts |cut -d"," -f1|uniq -c
10 1749
126 1785cat /tmp/counts |grep 1749
1749,SQ00015116
1749,SQ00015117
1749,SQ00015118
1749,SQ00015119
1749,SQ00015120
1749,SQ00015121
1749,SQ00015122
1749,SQ00015123
1749,SQ00015125
1749,SQ00015126In the future, we might want to reprocess these 10 plates and include the missing features
bethac07
commented
2 years ago This should make it possible if we need to - https://hub.docker.com/layers/cellprofiler/cellprofiler/2.3.1/images/sha256-d790b21623654e351390e283e3243860a7595120f7ba1e5f1df36b1277ea0cf1?context=explore
gwaybio
commented
2 years ago Pointing here https://github.com/broadinstitute/lincs-cell-painting/issues/3#issuecomment-591994451 as these plates seemed to have posed difficulties in the past.
@EchteRobert reported this:
Q about the LINCS dataset: I have run into plates which contain slightly fewer measured features than the bulk of the plates, i.e., 1745 instead of 1781. Is this a known issue? All of the plates that have 1745 features use platemap “C-7161-01-LM6-001”. See the thread for missing features. Note that these numbers are after preprocessing so especially some Image features may not be included in these 1781/1745
@bethac07 said:
I don't think it's a known issue but based on the error messages I see how it happened. Was likely a pilot batch and/or a batch that was rerun later would be my guess If you search Slack for the barcode names there might be a message about them (not certain but non zero)