search

aws / sagemaker-python-sdk

A library for training and deploying machine learning models on Amazon SageMaker
https://sagemaker.readthedocs.io/
Apache License 2.0
2.1k stars 1.14k forks source link

Narrowly specified dependencies and pointless docker_compose #1118

Closed jfelectron closed 4 years ago

jfelectron commented 4 years ago

Please fill out the form below.

System Information

Describe the problem

This is a very narrow specification that clashes with almost any other library that depends on requests, of which there are many. Requests is stable, I see no reason to pin it so narrowly. 

"requests>=2.20.0, <2.21"

This only appears to be needed for local testing and should not be required. It requires much older version of pyYaml and the docker python sdk (<4), which again clashes with many other libraries:

"docker-compose>=1.23.0"

The real root of the problem is that sagemaker uses:

pkg_resources.require("sagemaker")[0].version

This incurs very stringent runtime dependency conflict chekcing that is atypical for python libraries. If such strict depenendencies are really required, they should be enforced at install time and not runtime.

File "/Users/foleyjon/Documents/Repos/panem/panem/model/conversion_classifier.py", line 5, in import sagemaker File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/init.py", line 18, in from sagemaker import estimator, parameter, tuner # noqa: F401 File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/estimator.py", line 26, in from sagemaker.analytics import TrainingJobAnalytics File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/analytics.py", line 23, in from sagemaker.session import Session File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/session.py", line 35, in from sagemaker.user_agent import prepend_user_agent File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/user_agent.py", line 23, in SDK_VERSION = pkg_resources.require("sagemaker")[0].version File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/pkg_resources/init.py", line 900, in require needed = self.resolve(parse_requirements(requirements)) File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/pkg_resources/init.py", line 791, in resolve raise VersionConflict(dist, req).with_context(dependent_req) g_resources.ContextualVersionConflict: (docker 4.0.2 (/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages), Requirement.parse('docker[ssh]<4.0,>=3.7.0'), {'docker-compose'})

Minimal repro / logs

pip3 list

Package Version Location

appdirs 1.4.3 appnope 0.1.0 asn1crypto 1.1.0 atomicwrites 1.3.0 attrs 19.3.0 aws-cdk.assets 1.1.0 aws-cdk.aws-apigateway 1.1.0 aws-cdk.aws-applicationautoscaling 1.1.0 aws-cdk.aws-autoscaling 1.1.0 aws-cdk.aws-autoscaling-common 1.1.0 aws-cdk.aws-autoscaling-hooktargets 1.1.0 aws-cdk.aws-certificatemanager 1.1.0 aws-cdk.aws-cloudformation 1.1.0 aws-cdk.aws-cloudfront 1.1.0 aws-cdk.aws-cloudwatch 1.1.0 aws-cdk.aws-codebuild 1.1.0 aws-cdk.aws-codecommit 1.1.0 aws-cdk.aws-codedeploy 1.1.0 aws-cdk.aws-codepipeline 1.1.0 aws-cdk.aws-codepipeline-actions 1.1.0 aws-cdk.aws-ec2 1.1.0 aws-cdk.aws-ecr 1.1.0 aws-cdk.aws-ecr-assets 1.1.0 aws-cdk.aws-ecs 1.1.0 aws-cdk.aws-elasticloadbalancing 1.1.0 aws-cdk.aws-elasticloadbalancingv2 1.1.0 aws-cdk.aws-events 1.1.0 aws-cdk.aws-events-targets 1.1.0 aws-cdk.aws-iam 1.1.0 aws-cdk.aws-kms 1.1.0 aws-cdk.aws-lambda 1.1.0 aws-cdk.aws-logs 1.1.0 aws-cdk.aws-route53 1.1.0 aws-cdk.aws-route53-targets 1.1.0 aws-cdk.aws-s3 1.1.0 aws-cdk.aws-s3-assets 1.1.0 aws-cdk.aws-sagemaker 1.1.0 aws-cdk.aws-secretsmanager 1.1.0 aws-cdk.aws-servicediscovery 1.1.0 aws-cdk.aws-sns 1.1.0 aws-cdk.aws-sns-subscriptions 1.1.0 aws-cdk.aws-sqs 1.1.0 aws-cdk.aws-ssm 1.1.0 aws-cdk.aws-stepfunctions 1.1.0 aws-cdk.aws-stepfunctions-tasks 1.1.0 aws-cdk.core 1.1.0 aws-cdk.custom-resources 1.1.0 aws-cdk.cx-api 1.1.0 aws-cdk.region-info 1.1.0 awslogs 0.11.0 backcall 0.1.0 bcrypt 3.1.7 black 19.3b0 bleach 3.1.0 boto3 1.10.14 botocore 1.13.14 cached-property 1.5.1 cattrs 0.9.0 certifi 2019.9.11 cffi 1.13.2 chardet 3.0.4 Click 7.0 cloudpickle 1.2.2 cryptography 2.8 decorator 4.4.1 defusedxml 0.6.0 docker 4.0.2 docker-compose 1.24.1 docker-pycreds 0.4.0 dockerpty 0.4.1 docopt 0.6.2 docutils 0.15.2 entrypoints 0.3 fabric 2.5.0 flake8 3.7.8 flake8-blind-except 0.1.1 flake8-builtins 1.4.1 flake8-import-order 0.18.1 flake8-mypy 17.8.0 flake8-polyfill 1.0.2 idna 2.7 importlib-metadata 0.23 importlib-resources 1.0.2 invoke 1.3.0 ipykernel 5.1.3 ipython 7.9.0 ipython-genutils 0.2.0 ipywidgets 7.5.1 isort 4.3.18 jedi 0.15.1 Jinja2 2.10.1 jmespath 0.9.4 joblib 0.14.0 jsii 0.14.3 jsonschema 2.6.0 jupyter 1.0.0 jupyter-client 5.3.4 jupyter-console 6.0.0 jupyter-core 4.6.1 liquidpy 0.0.7 mantra 1.0.7.dev0 /Users/foleyjon/Documents/Repos/mantra MarkupSafe 1.1.1 mccabe 0.6.1 mistune 0.8.4 more-itertools 7.2.0 mypy 0.720 mypy-extensions 0.4.1 mysql-connector-python 8.0.17 nbconvert 5.6.1 nbformat 4.4.0 nominis 1.0.0 notebook 6.0.1 numpy 1.17.3 packaging 19.2 pandas 0.24.2 pandocfilters 1.4.2 paramiko 2.6.0 parso 0.5.1 pep8-naming 0.8.2 pexpect 4.7.0 pickleshare 0.7.5 pip 19.0.3 pluggy 0.13.0 prometheus-client 0.7.1 prompt-toolkit 2.0.10 protobuf 3.10.0 protobuf3-to-dict 0.1.5 psycopg2-binary 2.8.2 ptyprocess 0.6.0 publication 0.0.3 py 1.8.0 pycodestyle 2.5.0 pycparser 2.19 pyflakes 2.1.1 Pygments 2.4.2 PyNaCl 1.3.0 pyparsing 2.4.2 pyrsistent 0.15.5 pytest 5.0.1 pytest-clarity 0.1.0a1 pytest-mock 1.10.4 python-dateutil 2.8.0 pytz 2019.3 PyYAML 3.13 pyzmq 18.1.0 qtconsole 4.5.5 requests 2.20.1 s3transfer 0.2.1 sagemaker 1.43.4.post1 scikit-learn 0.21.3 scipy 1.3.2 Send2Trash 1.5.0 setuptools 41.6.0 six 1.13.0 sklearn 0.0 termcolor 1.1.0 terminado 0.8.2 testpath 0.4.2 texttable 0.9.1 toml 0.10.0 tornado 6.0.3 tqdm 4.37.0 traitlets 4.3.3 typed-ast 1.4.0 typing-extensions 3.7.4 urllib3 1.24.3 wcwidth 0.1.7 webencodings 0.5.1 websocket-client 0.56.0 wheel 0.33.6 widgetsnbextension 3.5.1 xgboost 0.90 zipp 0.6.0

jfelectron commented 4 years ago

In fact these are related problems that sagemaker itself faced. The solution here is clearly to drop docker-compose, which should be wholly unnecessary.

https://github.com/aws/sagemaker-python-sdk/commit/c042a6c880ef41e245346f2847ad14f9d6ce849a

nadiaya commented 4 years ago

Hi, thank you for the detail. docker-compose is needed for Local Mode, but I understand that it’s not needed for all Python SDK functionality.

We are aware of the issues caused by the current dependencies and we have plans on our roadmap to address them.

nadiaya commented 4 years ago

We moved docker-compose dependency under sagemaker[local] and it's no longer installed during regular installation. I hope this has solved your issue.