search

aws / amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
https://sagemaker-examples.readthedocs.io
Apache License 2.0
9.79k stars 6.67k forks source link

smddp_deepspeec_example doesn't run because of dependency issues. #4636

Closed jld23 closed 2 months ago

jld23 commented 2 months ago

Link to the notebook Add the link to the notebook.

Describe the bug Because of the pip dependencies in requirements.txt the example fails to run because of a mismatch with pydantic and deepspeed

The requirements below resolve the issue.

transformers==4.38.0
datasets
accelerate>=0.21
bitsandbytes
peft
deepspeed==0.14.0
Pydantic==1.10.15

To reproduce Run the example without modification and it will error.

Logs

Downloading deepspeed-0.9.2.tar.gz (779 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 779.3/779.3 kB 56.7 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      /opt/conda/lib/python3.10/site-packages/pydantic/_internal/_config.py:321: UserWarning: Valid config keys have changed in V2:
      * 'allow_population_by_field_name' has been renamed to 'populate_by_name'
      * 'validate_all' has been renamed to 'validate_default'
        warnings.warn(message, UserWarning)
      /opt/conda/lib/python3.10/site-packages/pydantic/_internal/_fields.py:149: UserWarning: Field "model_persistence_threshold" has conflict with protected namespace "model_".