search

crosszamirski / WS-DINO

WS-DINO: a novel framework to use weak label information in a self-supervised setting to learn phenotypic representations from high-content fluorescent images of cells.
19 stars 2 forks source link

CSV files required by illumination correction pipelines #2

Open hellowangqian opened 1 year ago

hellowangqian commented 1 year ago

Thanks a lot for sharing the code. I'm wondering if you can share more details of the illumination correction pipelines, and particularly it'll be much appreciated if you can provide the "loaddata.csv" and "LoadDataFile.csv" used in the pipelines. Thanks again and looking forward to your response.

crosszamirski commented 1 year ago

Hi @hellowangqian I believe these files are very standard and in standard format. We can't provide the exact files as they include mostly paths to our internal folder structure. However we can provide a template in the new year if that will be helpful!

hellowangqian commented 1 year ago

Thanks for your kind reply. A template with sensitive info removed would be very helpful. The other thing I'd like to ask is line298 in Eval_BBBC021.py, why dropping those rows (17 18 19 20 21 22 60 61 62 63 64) when calculating NSCB?

crosszamirski commented 1 year ago

@hellowangqian In the BBBC021 dataset there a some compounds/treatments which appear only in one batch hence we drop them from the NSCB evaluation as they have no matching MoA in a different batch - those are the rows in that sheet which correspond to those treatments.

hellowangqian commented 1 year ago

Brilliant! Another question on the combination of three channels: in which stage are they concatenated? image features or treatment features? before or after TVN?

发送自 Outlook for Androidhttps://aka.ms/AAb9ysg

From: Jan Cross-Zamirski @.> Sent: Wednesday, December 28, 2022 5:25:05 PM To: crosszamirski/WS-DINO @.> Cc: hellowangqian @.>; Mention @.> Subject: Re: [crosszamirski/WS-DINO] CSV files required by illumination correction pipelines (Issue #2)

@hellowangqianhttps://github.com/hellowangqian In the BBBC021 dataset there a some compounds/treatments which appear only in one batch hence we drop them from the NSCB evaluation as they have no matching MoA in a different batch

― Reply to this email directly, view it on GitHubhttps://github.com/crosszamirski/WS-DINO/issues/2#issuecomment-1366499596, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABX467XNU42ZDJZJFOLAKALWPQBPDANCNFSM6AAAAAATFG2JJM. You are receiving this because you were mentioned.Message ID: @.***>

crosszamirski commented 1 year ago

TVN is done by channel on the image level. Features are made for each channel then concatenated for the final evaluation

hellowangqian commented 1 year ago

Thanks for your valuable help, I now can reproduce similar results for unsupervised finetuning experiments, however, my results for weakly compound supervised training are far away from reported and are surprisingly worse than the unsupervised setting. The best NSCBs with 3 channels combined are around 0.83. Though the performance of three individual channels are better their counterparts without weakly supervised training. I basically use the default parameters hardcoded in the code for training. I wonder if you happen to have some clues why these unexpected performance happens and what mistakes I may have made. I myself do have some guesses:

  1. I notice that the csv file used in the WS-DINO_BBBC021.py script has a different name from that used in DINO_BBBC021.py. I just used the same csv file provided in the repo BBBC021_annotated.csv. Not sure if this is the issue.
  2. I am using a single GPU for the training, and the per-gpu batch size is 16, this makes the effective batch size half of your setting since you used two GPUs. I also tried increasing my batch size up to 32 but no improvement can be improved. I would appreciate it if you can provide some insights.

发送自 Outlook for Androidhttps://aka.ms/AAb9ysg

From: Jan Cross-Zamirski @.> Sent: Wednesday, December 28, 2022 6:49:52 PM To: crosszamirski/WS-DINO @.> Cc: hellowangqian @.>; Mention @.> Subject: Re: [crosszamirski/WS-DINO] CSV files required by illumination correction pipelines (Issue #2)

TVN is done by channel on the image level. Features are made for each channel then concatenated for the final evaluation

― Reply to this email directly, view it on GitHubhttps://github.com/crosszamirski/WS-DINO/issues/2#issuecomment-1366558370, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABX467TCT3SMPEMDV4W5JDLWPQLNBANCNFSM6AAAAAATFG2JJM. You are receiving this because you were mentioned.Message ID: @.***>

crosszamirski commented 1 year ago

Okay, hard to know exactly from the information you have given. the .csv file is just BBBC021_annotated.csv for both. It won't be because of 1 GPU. Are you using BBBC021 and the same labels?

Have you tried using the strong label (MoA/target - i.e. the same label you use in evaluating NSC/NSCB matching)? We found that this would very effectively and consistently push the NSC/NSCB close to 99% or even 100% within 50-100 epochs of training. If you can't reproduce that I would say there is a more fundamental problem with your implementation

crosszamirski commented 1 year ago

Also check our training logs and see how yours compare. We found that aggregating the channels usually always resulted in superior NSC and NSCB scores compared to the individual channels e.g. https://github.com/crosszamirski/WS-DINO/blob/main/training_logs/NSC_NSCB_Weak_Compound.csv

Finally, if you are using our scripts in Eval_BBBC021.py and TSNE.py, they are quite hand crafted to match the format of our data sheets. They are not universal functions so be careful using them out the box!

It sounds like it might be an aggregation/normalisation issue. How we did it will be in TSNE.py but I think we concatenated the features before any normalisation (L2) prior to evaluation with NSC/NSCB matching

hellowangqian commented 1 year ago

Thank you very much for clarifying things. I'm using BBBC021 and the same labels (i.e. those from this repository). Another thing that could vary is the illumination corrected images. Although I also used the cellprofiler pipeline files from this repository to get the illumination corrected images, I'm not sure if I have done it correctly. I got the csv file (consisting of 13200 rows for the whole list of images in BBBC021) from elsewhere to replace the missing "loaddata.csv " required by "BBBC021_IlluminationCalculate.cppipe" and a similar one (with added columns for specifying Illum.npy paths) for "BBBC021_IlluminationApply.cppipe". Thanks for the suggestion of using the strong label, I'm trying it a.t.m. I did check the training logs. My losses decrease slower than yours in general and end up with higher loss values for the same epochs. I have also tried TSNE.py to combine three channels, no luck (same results as my own implementation of channel combination).

crosszamirski commented 1 year ago

Hmm yeah does look like illumination correction could be a problem. We used a standard method as far as I am aware. let me know how you get on with MOA as the label. Also have a look at the self-attention maps, they could reveal a systematic illumination issue

hellowangqian commented 1 year ago

Hello @crosszamirski , here is some additional information that may be helpful to identify what the issue could be.

  1. I did the experiment with MoA as the pseudo weak labels and the results of NSCB are as follows. Each row represents an epoch from 0, 10, 20, ... to 390 and four columns represent DAPI, Tubulin, Actin and Combined, respectively. 

    [0.7391304347826086, 0.8260869565217391, 0.6413043478260869, 0.8478260869565217, 
0.6739130434782609, 0.8043478260869565, 0.6195652173913043, 0.7934782608695652,
0.717391304347826, 0.8152173913043478, 0.6413043478260869, 0.8043478260869565, 
0.6739130434782609, 0.8260869565217391, 0.6521739130434783, 0.7934782608695652, 
0.7065217391304348, 0.782608695652174, 0.6956521739130435, 0.7934782608695652, 
0.7391304347826086, 0.8260869565217391, 0.7065217391304348, 0.8369565217391305,
0.782608695652174, 0.8586956521739131, 0.7391304347826086, 0.9456521739130435, 
0.8043478260869565, 0.8913043478260869, 0.7608695652173914, 0.9565217391304348, 
0.8152173913043478, 0.9239130434782609, 0.7608695652173914, 0.9565217391304348, 
0.8152173913043478, 0.9130434782608695, 0.7717391304347826, 0.9891304347826086, 
0.8152173913043478, 0.9130434782608695, 0.8152173913043478, 0.9891304347826086, 
0.8260869565217391, 0.9239130434782609, 0.8152173913043478, 0.9891304347826086, 
0.8260869565217391, 0.9456521739130435, 0.8152173913043478, 0.9891304347826086, 
0.8260869565217391, 0.9347826086956522, 0.8260869565217391, 0.9891304347826086, 
0.8260869565217391, 0.9456521739130435, 0.8369565217391305, 0.9891304347826086,
0.8152173913043478, 0.9565217391304348, 0.8260869565217391, 0.9891304347826086, 
0.8260869565217391, 0.9456521739130435, 0.8369565217391305, 0.9891304347826086, 
0.8152173913043478, 0.9021739130434783, 0.8369565217391305, 0.9891304347826086, 
0.8152173913043478, 0.9347826086956522, 0.8369565217391305, 0.9891304347826086, 
0.8043478260869565, 0.9130434782608695, 0.8478260869565217, 0.9891304347826086, 
0.8152173913043478, 0.8913043478260869, 0.8478260869565217, 0.9891304347826086, 
0.8260869565217391, 0.8913043478260869, 0.8478260869565217, 0.9891304347826086, 
0.8369565217391305, 0.9021739130434783, 0.8478260869565217, 0.9891304347826086, 
0.8152173913043478, 0.9021739130434783, 0.8586956521739131, 0.9891304347826086, 
0.8369565217391305, 0.8695652173913043, 0.8369565217391305, 0.9891304347826086, 
0.8260869565217391, 0.8913043478260869, 0.8369565217391305, 0.9891304347826086, 
0.8369565217391305, 0.9021739130434783, 0.8586956521739131, 0.9891304347826086,
0.8260869565217391, 0.9130434782608695, 0.8260869565217391, 0.9891304347826086,
0.8260869565217391, 0.9347826086956522, 0.8369565217391305, 0.9891304347826086,
0.8369565217391305, 0.9021739130434783, 0.8478260869565217, 0.9891304347826086,
0.8478260869565217, 0.9130434782608695, 0.8260869565217391, 0.9891304347826086,
0.8260869565217391, 0.8913043478260869, 0.8260869565217391, 0.9891304347826086,
0.8369565217391305, 0.9021739130434783, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.9021739130434783, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.9239130434782609, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.9239130434782609, 0.8260869565217391, 0.9891304347826086,
0.8369565217391305, 0.9239130434782609, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.8913043478260869, 0.8260869565217391, 0.9891304347826086,
0.8369565217391305, 0.8913043478260869, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.8913043478260869, 0.8369565217391305, 0.9891304347826086]

  2. For compound weak label experiments, the results of NSCB are as follows. Each row represents an epoch from 0, 10, 20, ... to 190 and four columns represent DAPI, Tubulin, Actin and Combined, respectively. 

    [0.7391304347826086, 0.8043478260869565, 0.6304347826086957, 0.8478260869565217, 
0.6521739130434783, 0.8043478260869565, 0.6413043478260869, 0.75, 
0.6086956521739131, 0.75, 0.6086956521739131, 0.6847826086956522, 
0.6086956521739131, 0.8260869565217391, 0.5760869565217391, 0.6956521739130435, 
0.6086956521739131, 0.7934782608695652, 0.6195652173913043, 0.782608695652174, 
0.6304347826086957, 0.8043478260869565, 0.6195652173913043, 0.8369565217391305,
0.6413043478260869, 0.8804347826086957, 0.6521739130434783, 0.8478260869565217, 
0.6521739130434783, 0.9021739130434783, 0.6739130434782609, 0.8369565217391305, 
0.6630434782608695, 0.8695652173913043, 0.717391304347826, 0.8478260869565217, 
0.6630434782608695, 0.9130434782608695, 0.7065217391304348, 0.8478260869565217,
0.6956521739130435, 0.8913043478260869, 0.717391304347826, 0.8804347826086957, 
0.7065217391304348, 0.8913043478260869, 0.7065217391304348, 0.8913043478260869, 
0.6956521739130435, 0.8804347826086957, 0.7065217391304348, 0.8804347826086957, 
0.7065217391304348, 0.8695652173913043, 0.7282608695652174, 0.8586956521739131, 
0.7065217391304348, 0.8804347826086957, 0.7282608695652174, 0.8478260869565217, 
0.7065217391304348, 0.8586956521739131, 0.717391304347826, 0.8695652173913043, 
0.7065217391304348, 0.8695652173913043, 0.717391304347826, 0.8804347826086957, 
0.717391304347826, 0.8586956521739131, 0.7282608695652174, 0.8695652173913043,
0.7065217391304348, 0.8804347826086957, 0.717391304347826, 0.8913043478260869,
0.717391304347826, 0.8804347826086957, 0.717391304347826, 0.8913043478260869]

  3. Some examples of attention map visualisation using compounds as the weak labels. From top to bottom: DAPI image, DAPI attention, Tubulin image, Tubulin attention, Actin image, Actin attention. DAPI_img DAPI_attn-head5 Tubulin_img Actin_attn-head5 Actin_img Tubulin_attn-head5 /7327614/210293582-bc4dab9d-fe7a-4707-9b7e-43da89502a1f.png)

  4. Some observations: The combined results are not necessarily better than the best individual channel (usually the Tubulin channel) in my reproduced results; In the compound weak label experiments, the individual channel results seem better than your reported results for Tubulin and Actin but worse for DAPI. And the 3 channel combined results are much worse than your reported ones. Some thoughts: I still suspect the issue is in the illumination correction step since I basically use your code for other steps. It would be great to have your "loaddata.csv" and "LoadDataFile.csv" at your convenience for validation. Many thanks.

crosszamirski commented 1 year ago

nice work, cheers. It's only a small difference from our results which is good. If we used a different ilum correction that could explain it, I'm sure these things are quite sensitive to preprocessing steps. You could also just try re-running and see if you get the same (our were pretty consistent but worth checking).

did you just train for 190 epochs? I think our best model was 200 epochs but I would probably train for more than 190 to be sure.

Will try and get that loaddata file to you asap

hellowangqian commented 1 year ago

Hi @crosszamirski , thanks a million for your prompt reply. I understand the results for MoA weak labels are close to yours, however, the compound-as-weak-label results are far from satisfication. I will re-run this experiment with 400 epochs and update the results here though I doubt that will make a big difference. It would also be helpful if you can provide some exemplar files for validating the illumination correction results, e.g., a pair of images before and after applying the correction, and the corresponding "xxxIllum_xxx.npy" file.

crosszamirski commented 1 year ago

Yes going to need some time to get back to you on that so if you can find standard ilum correction elsewhere then try that. One more thing to help understand your results is plot tSNE maps, might reveal some insights

hellowangqian commented 1 year ago

A quick question: when you do the illumination correction, do you use all images in BBBC021 for calculating the Illumination Correction Functions or only the MoA-labelled subset of BBBC021?

crosszamirski commented 1 year ago

All of BBBC021. I have found our loaddata .csv files and they have the following headings: TableNumber ImageNumber Image_FileName_DAPI Image_PathName_DAPI Image_FileName_Tubulin Image_PathName_Tubulin Image_FileName_Actin Image_PathName_Actin Metadata_Plate Metadata_Well Image_FileName_DAPIIllum Image_FileName_ActinIllum Image_FileName_TubulinIllum