search

acharkq / ProtT3

Source code for ACL 2024 paper: "ProtT3: Protein-to-Text Generation for Text-based Protein Understanding"
48 stars 2 forks source link

The results using the checkpoint for stage1 do not match expectations. #3

Open pxy963 opened 2 months ago

pxy963 commented 2 months ago

Hello, I use your checkpoint to experiment on the retrieval for swiss prot and the proteinkg25. But I follow the guide in the readme, and I get the result is lower about 5% in the testset of the two datasets than the point in the paper. Is the checkpoint you provided the different from the results in the paper, or are there some experimental settings involved? Thank you.

acharkq commented 2 months ago

Hi, can you provide more details here?

pxy963 commented 2 months ago

I run this command 'python stage1.py --devices '0,1,2,3' --mode eval --filename stage1_ckpt --num_query_token 8 --plm_name "facebook/esm2_t30_150M_UR50D" --save_every_n_epochs 10 --batch_size 32 --precision 'bf16-mixed' --num_workers 8 --init_checkpoint /path/to/stage1.ckpt;' and I download your "stage1.ckpt" to test. For example, when retrieval in the test set of the swissprot, the T2P R@20 is 90.7, but in the paper is 95.8.