search

allenai / specter

SPECTER: Document-level Representation Learning using Citation-informed Transformers
Apache License 2.0
507 stars 55 forks source link

Fine-tuning SPECTER? #20

Open jacklxc opened 3 years ago

jacklxc commented 3 years ago

  1. Is there any way that instead of training from SciBERT, but directly fine-tune on SPECTER? It seems that the format of the model weights of SPECTER is different from SciBERT.

  2. How do I fine-tune SPECTER like SciBERT on classification tasks?

ZzyChris97 commented 2 years ago

I have the same problem, how did you solve this

armancohan commented 2 years ago

The model that is on huggingface should be easily fine-tunable like SciBERT. You can follow instructions here https://huggingface.co/docs/transformers/training but instead of bert-base-uncased use allenai/specter as the pre-trained model name.

gabriead commented 1 year ago

How does a custom training dataset has to look like? I understand from the repo that it consists of title+abstract+id for each paper in metadata.json but I don't understand was data.json does? Are those the positive and negative examples of papers for each paper?