Closed louieworth closed 3 months ago
Thank you for your interest in our work
Let me know if you have any more questions.
Thanks @vaibhavad
Does LLM2Vec support for other models, e.g., Pythia? I have checked it and it seems like it only supports llama and mistral families.
Just to confirm: if I want to train the model based on my dataset, the correct training code is on experiments/run_supervised.py
and I only need to adjust the "dataset_name": "E5"
.
Currently not, we plan to extend support to many more models in the future. If you are interested in contributing, feel free to open a PR, and I'll be happy to assist you
You'll need to add another file in the datasets folder, which specifies the loading/saving/instructions logic etc. If you are working on a fork, feel free to tag me and I'll be to assist better there.
@louieworth, the citation for [1] is not mentioned. Could you provide thr reference to it? I am interested in reading the mentioned paper.
I've updated the reference. Please check on it.
Thanks for your contribution to LLM2Vec.
Have you released the code training in MTEB (sequence-level tasks)? (I am not so sure whether the code between token-level and sequence-level tasks remains the same.)
Based on your modification to the decode-only Transformers, i.e., Llama, can I jointly train this model with supervised fine-tuning (SFT) via maximum likelihood? OR, the traditional SFT training is not available when training with LLM2Vec, and it must be the next stage of training for this model.
2.1 In training contrastive training approaches in other domains, i.e., CV. They typically first do the unsupervised contrastive learning and then freeze the features and train a supervised linear classifier (a fully connected layer followed by softmax) if the downstream tasks are classification [1]. Will it be the same in LLM2Vec? (similar to #50 )
[1] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.