niranjchandrasekaran
commented
1 year ago cc @shntnu
Open NinoDui opened 1 year ago
niranjchandrasekaran
commented
1 year ago cc @shntnu
NinoDui
commented
1 year ago Hey, I was wondering if there's any update about it? Thanks.
niranjchandrasekaran
commented
1 year ago Hi @NinoDui, I checked the load_data_with_illum.parquet and well.csv.gz files and found the same missing wells as you did.
@shntnu it looks like load_data.csv.gz has all 384 wells, but both load_data_with_illum.csv.gz and load_data_with_illum.parquet have only 70 wells.
NinoDui
commented
1 year ago Hi @NinoDui, I checked the
load_data_with_illum.parquetandwell.csv.gzfiles and found the same missing wells as you did.@shntnu it looks like
load_data.csv.gzhas all 384 wells, but bothload_data_with_illum.csv.gzandload_data_with_illum.parquethave only 70 wells.
@niranjchandrasekaran
Hey Niranj, thank you for all the support. Much appreciated it if we could figure it out. Thanks in advance :)
shntnu
commented
1 year ago Update: This comment ended up being off-topic but I am retaining it here because it is useful information
I'll copy over some notes from an internal thread and then respond in the next comment
[...]
Relatedly, there are wells for which there are images, but they don't appear in the metadata files. I think we have 9 of these wells:
plate well
36189 BR00121426 H14
39644 BR00121427 H14
43100 BR00121428 H14
53468 BR00121436 H14
598321 BR00125181 C10
663975 BR00125638 C10
665352 BR00125638 I19
948012 BR00127147 F16
948066 BR00127147 F22If you take that first one, for example, it is definitely the case that the files exist in the S3 buckets:
$ aws s3 ls s3://cellpainting-gallery/cpg0016-jump/source_4/images/2021_08_09_Batch11/images/BR00121426__2021-08-14T15_29_30-Measurement1/Images/ | grep r08c14 | head -5
2022-10-20 22:08:31 1853777 r08c14f01p01-ch1sk1fk1fl1.tiff
2022-10-20 22:05:06 1904709 r08c14f01p01-ch2sk1fk1fl1.tiff
2022-10-20 22:08:32 1712279 r08c14f01p01-ch3sk1fk1fl1.tiff
2022-10-20 22:08:32 1749803 r08c14f01p01-ch4sk1fk1fl1.tiff
2022-10-20 22:05:07 1816711 r08c14f01p01-ch5sk1fk1fl1.tiff...but it does not exist in the metadata manifest:
In [23]: wells[(wells.Metadata_Plate == "BR00121426") & (wells.Metadata_Well == "H14")]
Out[23]:
Empty DataFrame
Columns: [Metadata_Source, Metadata_Plate, Metadata_Well, Metadata_JCP2022]
Index: []On further probing, it turned out that the wells were blank.
shntnu
commented
1 year ago As noted in https://github.com/jump-cellpainting/datasets/issues/61#issuecomment-1499094909, if a well is empty, the images will exist but no profiles will be created, and thus there will be no entry in wells.csv.gz
I'll add this issue to our FAQ #62
niranjchandrasekaran
commented
1 year ago As noted in https://github.com/jump-cellpainting/datasets/issues/61#issuecomment-1499094909, if a well is empty, the images will exist but no profiles will be created, and thus there will be no entry in wells.csv.gz
IIUC, the problem seems to be the other way around. For this plate, there are wells and in well.csv.gz and the aggregated profile parquet file also has all 384 wells. But the load_data_with_illum files seem to be missing wells.
shntnu
commented
1 year ago Sorry, I totally missed that.
it looks like
load_data.csv.gzhas all 384 wells, but bothload_data_with_illum.csv.gzandload_data_with_illum.parquethave only 70 wells.
load_data.csv.gz seems to have the same number of rows as load_data_with_illum.csv.gz. can you verify @niranjchandrasekaran ?
aws s3 cp s3://cellpainting-gallery/cpg0016-jump/source_10/workspace/load_data_csv/2021_08_12_U2OS_48_hr_run15/Dest210803-160702/load_data.csv.gz -|gunzip - |wc -l
# 417
aws s3 cp s3://cellpainting-gallery/cpg0016-jump/source_10/workspace/load_data_csv/2021_08_12_U2OS_48_hr_run15/Dest210803-160702/load_data_with_illum.csv.gz -|gunzip - |wc -l
# 417That said, I confirmed that the images folder does have all 384 wells
aws s3 ls s3://cellpainting-gallery/cpg0016-jump/source_10/images/2021_08_12_U2OS_48_hr_run15/images/Dest210803-160702/ |gzip > ~/Desktop/source_10_2021_08_12_U2OS_48_hr_run15_Dest210803-160702.txt.gz
gzcat ~/Desktop/source_10_2021_08_12_U2OS_48_hr_run15_Dest210803-160702.txt.gz|grep tif$|grep Dest21|tr -s " "|cut -d" " -f4|cut -d"_" -f2|sort|uniq -c|wc -l
# 384
gzcat ~/Desktop/source_10_2021_08_12_U2OS_48_hr_run15_Dest210803-160702.txt.gz|grep tif$|grep Dest21|tr -s " "|cut -d" " -f4|cut -d"_" -f2|sort|uniq -c|cut -d" " -f3|sort |uniq -c
# 384 48load_data.csv.gz seems to have the same number of rows as load_data_with_illum.csv.gz. can you verify @niranjchandrasekaran ?
You are right. load_data.csv.gz also has the same number of wells as the two illum files.
shntnu
commented
1 year ago @NinoDui Thank you so much for flagging this! Can you help us report how prevalent this issue is?
Here's how you'd do it
import pandas as pd
source_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13]
df_all = pd.DataFrame()
for source_id in source_ids:
df = load(
dataset="cpg0016-jump",
source=f"source_{source_id}}",
component="load_data_csv",
batch="2021_06_14_Batch6",
plate="BR00121429",
columns=["Metadata_Source", "Metadata_Plate", "Metadata_Well"],
)
# keep only distinct rows
df = df.drop_duplicates()
df_all = df_all.append(df)Now report which wells are present in well.csv.gz but absent in df_all
Warning: I haven't checked the code (that was autogenerated by Copilot :D)
NinoDui
commented
1 year ago I've just checked with my own comparison script and found 125 mismatches (at plate level) between the number of wells available on a plate and those provided by corresponding load_data_with_illum.parquet.
The result is attached.
plate.csv.gz and wells.csv.gzload_data_with_illum.parquetHow the wells from parquet are counted:
S3_LOADDATA_FORMATTER = (
"s3://cellpainting-gallery/cpg0016-jump/"
"{Metadata_Source}/workspace/load_data_csv/"
"{Metadata_Batch}/{Metadata_Plate}/load_data_with_illum.parquet"
)
def fetch_well_from_parquet(row: pd.Series):
meta_path = S3_LOADDATA_FORMATTER.format(**row)
meta = pd.read_parquet(meta_path, storage_options=REMOTE_STORAGE_OPTION)
wells_from_parquet = meta['Metadata_Well'].unique()
return wells_from_parquet.shape[0]How the number of wells from plate & well metas are calculated:
wells_info = wells.merge(plates, on=['Metadata_Source', 'Metadata_Plate'])
wells_info = wells_info[["Metadata_Source", "Metadata_Batch", "Metadata_Plate", "Metadata_Well"]]
well_count = wells_info.groupby(['Metadata_Source', 'Metadata_Batch', 'Metadata_Plate'])\
.agg(n_well_on_plate=('Metadata_Well', pd.Series.count))@NinoDui Thank you so much for flagging this! Can you help us report how prevalent this issue is?
Here's how you'd do it
import pandas as pd source_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13] df_all = pd.DataFrame() for source_id in source_ids: df = load( dataset="cpg0016-jump", source=f"source_{source_id}}", component="load_data_csv", batch="2021_06_14_Batch6", plate="BR00121429", columns=["Metadata_Source", "Metadata_Plate", "Metadata_Well"], ) # keep only distinct rows df = df.drop_duplicates() df_all = df_all.append(df)Now report which wells are present in
well.csv.gzbut absent indf_allWarning: I haven't checked the code (that was autogenerated by Copilot :D)
shntnu
commented
1 year ago This is so helpful! Thank you very much.
Looking at your CSV file, I am relived to note that the only issue across the entire dataset is with that one plate you reported originally. For all others, there are more wells with images than with profiles, and this can happen, as mentioned in one of my previous comments above.
We will recreate the load data parquet for that plate.
This might take a while until we get to it. Will that block you?
Thanks again!
April 2024 Update: I took the CSV in https://github.com/jump-cellpainting/datasets/issues/61#issuecomment-1508083098 and turned it into a table below for easy searching.
NinoDui
commented
1 year ago Cool! Glad to know the issue led to limited effect. Thanks for all the checking and effort behind.
I am taking on DL Model experiments based on the images and I could exclude the issued one manually. That's not blocking for short.
Still hope to hear the issue be settled and benefit a larger group of researchers. You're actually doing something not only meaningful but cool.
Best regards, NinoDui
This is so helpful! Thank you very much.
Looking at your CSV file, I am relived to note that the only issue across the entire dataset is with that one plate you reported originally. For all others, there are more wells with images than with profiles, and this can happen, as mentioned in one of my previous comments above.
We will recreate the load data parquet for that plate.
This might take a while until we get to it. Will that block you?
Thanks again!
Hi there,
I happened to find the metadata for wells of source_10, batch 2021_08_12_U2OS_48_hr_run15, and plate Dest210803-160702 may be missed. May I get your help in double-checking it? Or feel free to correct me if I am not taking it at the right pace.
A quick demo of how to revise it:
It seems like the well info provided by
wells.csv.gzis larger in amount compared to those retrieved fromload_data_with_illum.parquet. Is that a corner case that I missed? Or is that being uploaded in progress?Thanks for your time and effort.
Best wishes, Nino