Has chaoyi-wu/PMC_LLAMA_7B checkppint gone through an SFT or RHLF ?

chaoyi-wu / PMC-LLaMA

The official codes for "PMC-LLaMA: Towards Building Open-source Language Models for Medicine"

549 stars 52 forks source link

Has chaoyi-wu/PMC_LLAMA_7B checkppint gone through an SFT or RHLF ? #30

Open shamanez opened 5 months ago

shamanez commented 5 months ago

It sits sIs the above checkpoint the pre-trained model with unsupervised data and hasn't seen any instruction-tuning datasets?

I have another question: did you use llama2-base as the base model to conduct continual pre-training with research papers and books?

chaoyi-wu commented 3 months ago

No, PMC_LLaMA_7B has not undergone any instruction tunining datasets while our latest PMC_LLaMA_13B has been instruction tuned.

We have not tried LLaMA-2 for continual pre-training, since in our evaluation, LLaMA-2, compared with LLaMA, is only enhanced with instruction following ability and in basic knowledge, the gain is limited.