Fine tuning: Some weights of the model checkpoint were not used when initializing T5EncoderModel

[cayden-uw]

Thanks for open sourcing this model! I have attempted to fine tune instructor-large and instructor-xl.

I now get an error on the outputted model when I try to load it "Some weights of the model checkpoint at ../up-l/ were not used when initializing T5EncoderModel" similarly the outputted model is missing key files like modules.json etc. which I attempted to use from your existing model given it should be the same architecture.

I modified medi-data.json to have my training data.

[hongjin-su]

Hi, Thanks a lot for your interest in the INSTRUCTOR model!

Could you help to provide the codes to save and load the output models?

[cayden-uw]

I was using your train.py and loading as per your model INSTRUCTOR(path) just with the path of the output from train.py (with the addition of the modules.json etc. I copied from the existing instructor model).

Thanks for the quick reply and your help!

[hongjin-su]

Have you cloned and installed the latest repo and the InstructorEmbedding package? The latest InstructorEmbedding package is in the version 1.0.1.

[cayden-uw]

I was using 1.0.0 and had the latest repo cloned.

Steps to repro:

1. Create cache folder. Add in custom media-data.sjon
2. Create output folder
3. python train.py --model_name_or_path hkunlp/instructor-large --output_dir {output_directory} --cache_dir {cache_directory} --max_source_length 512 --num_train_epochs 10 --save_steps 500 --cl_temperature 0.01 --warmup_ratio 0.1 --learning_rate 2e-5

The only thing I changed was a small modification to the train script to make checkpoints load

[hongjin-su]

You may update the InstructorEmbedding package to the latest version 1.0.1.

Feel free to add any further questions or comments!

[hongjin-su]

Please re-open the issue if you have any questions or comments!