search

ibrahimethemhamamci / CT-CLIP

Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
197 stars 21 forks source link

Different normalization processing during the validation process and training process #36

Open jackhu-bme opened 1 week ago

jackhu-bme commented 1 week ago

The normalization range during training and inference is different. During training, [-1000, 1000] -> [-1, 1]. while during inference, [-1000, 200] -> [-1, 1]. Why? This might cause performance drop severely!

Based on: The release code of scripts/data.py, in nii_img_to_tensor function of CTReportDataset, the range is [-1000, 1000]. However, the release code of scripts/data_inference.py, in nii_img_to_tensor function of CTReportDatasetInfer, the range is [-1000, 200].

Please tell us why, or we are hard to follow this setting. We believe consistent pre-processing between train and infer is very very important. Seem like a little but serious mistake here!

jackhu-bme commented 1 week ago

look: https://github.com/ibrahimethemhamamci/CT-CLIP/blob/a00eceb016c7a1feb2256f0dde7c2b27fd7a6464/scripts/data.py#L132 and https://github.com/ibrahimethemhamamci/CT-CLIP/blob/a00eceb016c7a1feb2256f0dde7c2b27fd7a6464/scripts/data_inference.py#L81

jackhu-bme commented 1 week ago

@ibrahimethemhamamci @sezginerr

sezginerr commented 1 week ago

Hi @jackhu-bme,

We have decided to change the clipping range from [-1000,200] to [-1000,1000] include not lung related parts of the CT, such as bones, in the CT-CLIP v2. This was necessary for the subsequent CT-CHAT training (bones etc. are also evaluated in the reports.) Apparently, I did not update the inference dataset script when I cleaned the code for the second iteration (see zero_shot.py, where the imported dataset script name is data_inference_nii, not data_inference). I will send the new script, but inference is also done between -1000 and 1000. Thanks for pointing this out!

jackhu-bme commented 1 week ago

Thanks a lot! I now understand what's going on. Thanks for telling us the reason and fix this error. By the way, maybe you could also update the arxiv of CT-CLIP to tell us the clipping range is changed. I actually believe that [-1000, 1000] is a better range according to my experience on CT processing and analysis.

jackhu-bme commented 1 week ago

Hi @jackhu-bme,

We have decided to change the clipping range from [-1000,200] to [-1000,1000] include not lung related parts of the CT, such as bones, in the CT-CLIP v2. This was necessary for the subsequent CT-CHAT training (bones etc. are also evaluated in the reports.) Apparently, I did not update the inference dataset script when I cleaned the code for the second iteration (see zero_shot.py, where the imported dataset script name is data_inference_nii, not data_inference). I will send the new script, but inference is also done between -1000 and 1000. Thanks for pointing this out!

By the way, could you please tell us about your new work CT-CHAT? is there an arxiv paper or github repo?

sezginerr commented 1 week ago

Hi @jackhu-bme, we have updated the preprint recently. Please see: https://arxiv.org/pdf/2403.17834 for the updated preprint and https://github.com/ibrahimethemhamamci/CT-CHAT for GitHub repo.

jackhu-bme commented 1 week ago

Thanks a lot! I will follow your work!