search

ibrahimethemhamamci / CT-CLIP

Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
200 stars 21 forks source link

Issue Training LiPro #26

Closed mz0g closed 3 months ago

mz0g commented 3 months ago

I am trying to finetune the LiPro model but I keep running into this "mismatch" issue related to the keys. Is the model I downloaded different to the one that needs to be loaded in? Not really sure how to proceed. I've tried redownloading the model and reseting my environments and recloning the whole repository.

Screenshot 2024-08-01 at 10 57 35 AM

sezginerr commented 3 months ago

Hi @mz0g . This should be related with transformers version. Can you make sure you are using transformers==4.30.1?

mz0g commented 3 months ago

Yes, I am using transformers==4.30.1. I think the issue is related to the fact that the loaded model's state dictionary adds trained_model to the front of each key, while the state_expect does not expect that. I made thid change and it seemed to fix the problem but now I am running into another issue.

mz0g commented 3 months ago

My number of samples is 0, so I think there is an issue with how I am loading my dataset. My current data folder points to /train_preprocessed which contains a list of subdirs train_001.nii.gz/train_case001.nii.gz/case_001.npz. Should VolumeName be train_case001.nii.gz?

mz0g commented 3 months ago

Looks like I fixed the issue once I set up my train_processed as follows: train_case_001.nii.gz/train_case_001.nii.gz/train_case_001.npz with VolumeName being train_case_001.nii.gz (each case has one corresponding report).