Add a way to instantiate from HF.AutoModel (again)

[dmitrii-palisaderesearch]

Hi,

I was looking to run simple_evaluate with my own transformers model and found #521 / #601, but it looks like they're only merged to master. Was this feature lost in the 0.4.0 migration? Should I port it?

[haileyschoelkopf]

Hi! This should be supported via the HF model type in main as well -- https://github.com/EleutherAI/lm-evaluation-harness/blob/f257d38b66f00f544aab5817fe66eb9fe78a8b0f/lm_eval/models/huggingface.py#L81 and we have people who use this feature. Does using this work for you?

[dmitrii-palisaderesearch]

Ohh, thank you, I was confused by the API there.

For anyone who was confused like me, it's

model = transformers.AutoModelForCausalLM.from_pretrained(...)
lm_eval.simple_evaluate(
    model=lm_eval.models.huggingface.HFLM(model)
)

And here's what you would do to run inference with unsloth:

from unsloth import FastLanguageModel
import lm_eval

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit",
    max_seq_length = 8192,
    load_in_4bit = True,
)

results = lm_eval.simple_evaluate(
    model=lm_eval.models.huggingface.HFLM(
        pretrained=model,
        tokenizer=tokenizer,
    ),
)

[abdalgader-a]

@dmitrii-palisaderesearch - have you run the script via accelerate ?

[dmitrii-palisaderesearch]

Nope, the above was plain Colab. I believe you can leverage accelerate by passing FastLanguageModel.from_pretrained(device_map=XXX) though.