search

aws / amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
https://sagemaker-examples.readthedocs.io
Apache License 2.0
9.8k stars 6.67k forks source link

[Bug Report] All fine tunes for Mistral 7b using sagemaker jumpstart are currently failing. #4553

Open aadupirn opened 5 months ago

aadupirn commented 5 months ago

All fine tunes for Mistral 7b using sagemaker jumpstart are currently failing with:

"ImportError: cannot import name 'insecure_hashlib' from 'huggingface_hub.utils' (/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/init.py)"

After changing nothing on my end fine tuning started to fail after many previous successful fine tunes.

To reproduce Try to fine tune any model using the guides here: https://aws.amazon.com/blogs/machine-learning/fine-tune-and-deploy-mistral-7b-with-amazon-sagemaker-jumpstart/

aadupirn commented 5 months ago

I was able to work around this:

The bug is still in sagemaker but I was able to workaround it but downloading and unpacking the source from AWS that is used and manually fixing deps by building new linux whl's that don't break. Using the latest whls for linux 2014 for transformers, tokenizer and huggingface_hub will fix it.

tuneshverma commented 5 months ago

I am also facing the same issue, can you please elaborate more, how you fixed the issue. I am fairly new to this. Did you get the sagemaker image and create a container and then did the required change and created new image from that and used that, if yes then from where did you get the image and how to use the new image with sagemaker train pipeline. Thanks in advance.

tuneshverma commented 5 months ago

solved the issue by changing the version of transformers to latest which is 4.37.2, which updated the tokenizer and huggingface_hub