Deploying evaluation for finetuned model as AWS SM pipeline step

[Avistian]

Hello,

I was wondering if there is possibility to incorporate lighteval into SM Pipelines. We are having custom fine-tuning pipeline that uses HuggingFace estimator to fine-tune the model in TrainingStep. Now, we would like to evaluate the fine-tuned model using your library, however I have seen that there is only option to either pass endpoint name with some additional config, or you can only evaluate base models from the HF Hub.

Therefore I have two questions:

1. Is there a way to evaluate fine-tuned model without first deploying it? I assume no, but still worth asking I guess ;)
2. If no, do you maybe know if there is a way to deploy the model inside the pipeline? I have seen recently the blogpost(https://www.philschmid.de/sagemaker-evaluate-llm-lighteval), however I would like to use this method in the pipeline itself. Is there a way to do it in such way? I have only seen its possible through Lambda step, however I would love to double check if there is any recommended way from your side to tackle that issue.

Thanks, Piotr

[clefourrier]

Hi!

Thanks for your interest!

It is at the moment possible to evaluate models in 3 ways:

• by deploying an inference endpoint with your model
• locally using transformers/accelerate, if your model is AutoModelForCausalLM.from_pretrained compatible
• locally using nanotron (likely not your use case)

So it is possible to evaluate models without first deploying them as long as they are transformers compatible.

I've never used SM Pipelines so I'm unsure about the rest of your question, feel free to add more information about your use case.

Best, Clémentine

[clefourrier]

Closing for inactivity