search

ncbi / BioREx

25 stars 9 forks source link

Loaded 0 parameters in the TF 2.0 model. Some weights of the model were not used when initializing the TF 2.0 model TFBertForSequenceClassification #2

Open lou-yimin opened 6 months ago

lou-yimin commented 6 months ago

I tried to run ’scripts/run_biorex_exp.sh‘ to reproduce your results, but ran into the following trouble: When I test directly with microsoft pre-trained models, the warning prompts:

[INFO|modeling_tf_utils.py:2830] 2024-02-29 20:59:44,616 >> loading weights file microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract/pytorch_model.bin [INFO|modeling_tf_pytorch_utils.py:184] 2024-02-29 20:59:45,290 >> Loading PyTorch weights from /root/autodl-tmp/BioREx/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract/pytorch_model.bin [INFO|modeling_tf_pytorch_utils.py:187] 2024-02-29 20:59:45,855 >> PyTorch checkpoint contains 133,577,846 parameters [INFO|modeling_tf_pytorch_utils.py:344] 2024-02-29 20:59:46,677 >> Loaded 0 parameters in the TF 2.0 model.

When I run the ‘scripts/run_biorex_exp.sh’ and predict using the model I got from my own training, it warns again. No parameters are loaded in and the result is really poor：

Loaded 0 parameters in the TF 2.0 model.

But when I use your pre-trained weights to predict directly it is normal.

loading weights file pretrained_model_biolinkbert/tf_model.h5 [WARNING|modeling_tf_utils.py:2966] 2024-02-29 20:55:57,942 >> All model checkpoint layers were used when initializing TFBertForSequenceClassification.

[WARNING|modeling_tf_utils.py:2975] 2024-02-29 20:55:57,942 >> All the layers of TFBertForSequenceClassification were initialized from the model checkpoint at pretrained_model_biolinkbert. If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForSequenceClassification for predictions without further training.

ptlai commented 4 months ago

Hi @lou-yimin,

My apologies for the delayed response. If you are still interested in BioREx but are experiencing this issue. Could you please provide more information about this issue? For example, your environment and the complete output log while running the program. Please feel free to email me (laip2@nih,gov) with any questions you may have. Thank you very much.