search

socialfoundations / lawma

Lawma: A lightly fine-tuned Llama model for legal classification tasks.
https://arxiv.org/abs/2407.16615
6 stars 0 forks source link

Lawma-8b and Lawma-70B huggingface model missing tokenizer #1

Closed Yixing-Li closed 1 month ago

Yixing-Li commented 1 month ago

Thanks for your work. But there seems some file missing:

  1. Seems missing Tokenizer on huggingface, https://huggingface.co/ricdomolm/lawma-8b
  2. Please explain the package used in evaluation hf_eval.py from llm_utils import load_tokenizer_model, etc There seems no .py named llm_utils and llm_utils package seems don't have load_tokenizer_model function pip install llm_utils image
Yixing-Li commented 1 month ago

Not a big deal with this problem. I've got tokenizer from the official llama3-8b-instruction huggingface. But probably you could add tokenizer to your huggingface :)

RicardoDominguez commented 1 month ago

Thanks! Just added the tokenizer. I'm also adding the llm_utils file now.