aws / sagemaker-python-sdk

A library for training and deploying machine learning models on Amazon SageMaker
https://sagemaker.readthedocs.io/
Apache License 2.0
2.07k stars 1.12k forks source link

Runtime dependency on setuptools from importing distutils #4534

Open jmahlik opened 3 months ago

jmahlik commented 3 months ago

Describe the bug Distutils was removed from the standard library in python 3.12. Environments created under 3.12 no longer contain setuptools by default. Distutils itself is also deprecated.

There's a few usages of distutils in the sagemaker python sdk. They are easily replaced by shutil. I do see a comment in sagemaker.workflow._repack_model about not wanting to use shutil. But with the dirs_exist_ok=True parameter added in python 3.8, it wouldn't fail on existing directories. So all supported python versions now have an alternative.

https://github.com/aws/sagemaker-python-sdk/blob/e163807110924fc0c63ca236848507702fd30ded/src/sagemaker/workflow/_repack_model.py#L30-L35

git grep -e distutils
.pylintrc:ignored-modules=distutils
src/sagemaker/local/image.py:from distutils.spawn import find_executable
src/sagemaker/local/utils.py:from distutils.dir_util import copy_tree
src/sagemaker/local/utils.py:    """A wrapper around distutils.dir_util.copy_tree.
src/sagemaker/workflow/_repack_model.py:# distutils.dir_util.copy_tree works way better than the half-baked
src/sagemaker/workflow/_repack_model.py:from distutils.dir_util import copy_tree
tests/data/_repack_model.py:# distutils.dir_util.copy_tree works way better than the half-baked
tests/data/_repack_model.py:from distutils.dir_util import copy_tree
tests/unit/test_chainer.py:from distutils.util import strtobool

I'll submit a PR to remove the deprecated calls in the source.

To reproduce A clear, step-by-step set of instructions to reproduce the bug. The provided code need to be complete and runnable, if additional data is needed, please include them in the issue.

# No setuptools installed by default
python3.12 -m venv
# or virtualenv venv --py 3.12
. venv/bin/activate
# Install from current master
python -m pip install -e .[local]
python -c 'import sagemaker.local.image'
...
    from distutils.spawn import find_executable
  ModuleNotFoundError: No module named 'distutils'

Expected behavior Imports do not fail.

Screenshots or logs If applicable, add screenshots or logs to help explain your problem.

System information A description of your system. Please provide:

Additional context Add any other context about the problem here.

eflanagan0 commented 1 week ago

Thank you for taking the time to document and attempt fixing this @jmahlik!

As a new user of AWS SageMaker, I am running into this bug when attempting to deploy a model using the SageMaker SDK.

eflanagan0 commented 1 week ago

relates to #3028