search

huggingface / notebooks

Notebooks using the Hugging Face libraries 🤗
Apache License 2.0
3.54k stars 1.5k forks source link

training, deploying sentence transformer model ( setfit method) in sagemaker pipeline #439

Open miteshkotak opened 10 months ago

miteshkotak commented 10 months ago

error i face while training sentence transformer model using "SetFitTrainer" and importing model with "SetfitModel" throws following error:

Traceback (most recent call last): File "train-setfit.py", line 81, in from setfit import SetFitModel File "/opt/conda/lib/python3.8/site-packages/setfit/init.py",

line 5, in from .trainer import SetFitTrainer File "/opt/conda/lib/python3.8/site-packages/setfit/trainer.py", line 4, in import evaluate File "/opt/conda/lib/python3.8/site-packages/evaluate/init.py", line 29, in from .evaluation_suite import EvaluationSuite File "/opt/conda/lib/python3.8/site-packages/evaluate/evaluation_suite/init.py", line 7, in from datasets import Dataset, DownloadMode, load_dataset | Traceback (most recent call last): File "train-setfit.py", line 81, in from setfit import SetFitModel File "/opt/conda/lib/python3.8/site-packages/setfit/init.py", line 5, in from .trainer import SetFitTrainer File "/opt/conda/lib/python3.8/site-packages/setfit/trainer.py", line 4, in import evaluate File "/opt/conda/lib/python3.8/site-packages/evaluate/init.py", line 29, in from .evaluation_suite import EvaluationSuite File "/opt/conda/lib/python3.8/site-packages/evaluate/evaluation_suite/init.py", line 7, in from datasets import Dataset, DownloadMode, load_dataset

Is it possible to run training job in sagemaker pipeline to train using Setfit method?

miteshkotak commented 10 months ago

Tagging you both here @philschmid @nickprock, as i think both of might have quick feedback for me. Many thanks in advance :)

nickprock commented 10 months ago

Hi @miteshkotak , it's hard to tell from this error for me. I used sagemaker library only for deploying from S3 never for train but I think it can be done. Check this notebook in cell 10. I think you need the requirements.txt file to import setfit to do the training of custom models. @philschmid will surely be able to tell you more.