aws / sagemaker-experiments

Experiment tracking and metric logging for Amazon SageMaker notebooks and model training.
Apache License 2.0
126 stars 36 forks source link

lowercase source arn before listing trial components #172

Closed danabens closed 1 year ago

danabens commented 1 year ago

Issue #, if available: internal customer ticket

Description of changes: near term mitigation for arn casing changes on backend which are currently causing Tracker.load to fail when a training job has upper case.

Testing done: locally `tox -e p38 -- -m 'not docker'

❯ tox -e py38 -- -m 'not docker'
GLOB sdist-make: /Users/danabens/workplace/github/aws/sagemaker-experiments/setup.py
py38 inst-nodeps: /Users/danabens/workplace/github/aws/sagemaker-experiments/.tox/.tmp/package/1/sagemaker-experiments-0.1.40.dev3+g954d731.zip
py38 installed: atomicwrites==1.4.1,attrs==22.1.0,boto3==1.26.16,botocore==1.29.16,certifi==2022.9.24,charset-normalizer==2.1.1,coverage==6.5.0,distlib==0.3.6,docker==6.0.1,exceptiongroup==1.0.4,execnet==1.9.0,filelock==3.8.0,flake8==6.0.0,idna==3.4,importlib-metadata==0.23,iniconfig==1.1.1,jmespath==1.0.1,joblib==1.2.0,mccabe==0.7.0,more-itertools==9.0.0,numpy==1.23.5,packaging==21.3,pandas==1.5.2,platformdirs==2.5.4,pluggy==0.13.1,py==1.11.0,pycodestyle==2.10.0,pyflakes==3.0.0,pyparsing==3.0.9,pytest==4.4.1,pytest-cov==2.9.0,pytest-cover==3.0.0,pytest-coverage==0.0,pytest-forked==1.4.0,pytest-rerunfailures==8.0,pytest-xdist==1.34.0,python-dateutil==2.8.2,pytz==2022.6,requests==2.28.1,s3transfer==0.6.0,sagemaker-experiments @ file:///Users/danabens/workplace/github/aws/sagemaker-experiments/.tox/.tmp/package/1/sagemaker-experiments-0.1.40.dev3%2Bg954d731.zip,scikit-learn==0.24.2,scipy==1.9.3,six==1.16.0,sklearn==0.0.post1,threadpoolctl==3.1.0,toml==0.10.2,tomli==2.0.1,tox==3.13.1,urllib3==1.26.12,virtualenv==20.16.7,websocket-client==1.4.2,zipp==3.10.0
py38 run-test-pre: PYTHONHASHSEED='2266544003'
py38 run-test: commands[0] | coverage run --source smexperiments -m pytest -m 'not docker'
=================================================================================================================================================== test session starts ====================================================================================================================================================
platform darwin -- Python 3.8.13, pytest-4.4.1, py-1.11.0, pluggy-0.13.1
cachedir: .tox/py38/.pytest_cache
rootdir: /Users/danabens/workplace/github/aws/sagemaker-experiments, inifile: pytest.ini
plugins: cov-2.9.0, forked-1.4.0, rerunfailures-8.0, xdist-1.34.0
collected 194 items / 1 deselected / 193 selected

tests/integ/test_experiment.py ...........                                                                                                                                                                                                                                                                           [  5%]
tests/integ/test_search_expression.py .s.                                                                                                                                                                                                                                                                            [  7%]
tests/integ/test_track_from_processing_job.py s                                                                                                                                                                                                                                                                      [  7%]
tests/integ/test_track_from_training_job.py s                                                                                                                                                                                                                                                                        [  8%]
tests/integ/test_tracker.py ..........                                                                                                                                                                                                                                                                               [ 13%]
tests/integ/test_training_job.py s                                                                                                                                                                                                                                                                                   [ 13%]
tests/integ/test_trial.py ........                                                                                                                                                                                                                                                                                   [ 18%]
tests/integ/test_trial_component.py ..........                                                                                                                                                                                                                                                                       [ 23%]
tests/unit/test_api_types.py ...                                                                                                                                                                                                                                                                                     [ 24%]
tests/unit/test_base_types.py ............                                                                                                                                                                                                                                                                           [ 31%]
tests/unit/test_environment.py .....                                                                                                                                                                                                                                                                                 [ 33%]
tests/unit/test_experiment.py .................                                                                                                                                                                                                                                                                      [ 42%]
tests/unit/test_list_trial_components_from_trial.py ......                                                                                                                                                                                                                                                           [ 45%]
tests/unit/test_metrics.py ..........                                                                                                                                                                                                                                                                                [ 50%]
tests/unit/test_search_expression.py .....                                                                                                                                                                                                                                                                           [ 53%]
tests/unit/test_tracker.py .....................................................                                                                                                                                                                                                                                     [ 80%]
tests/unit/test_training_job.py .                                                                                                                                                                                                                                                                                    [ 81%]
tests/unit/test_trial.py ...................                                                                                                                                                                                                                                                                         [ 91%]
tests/unit/test_trial_component.py ............                                                                                                                                                                                                                                                                      [ 97%]
tests/unit/test_utils.py .....                                                                                                                                                                                                                                                                                       [100%]

===================================================================================================================================================== warnings summary =====================================================================================================================================================
tests/unit/test_tracker.py::test_log_pr_curve
  /Users/danabens/workplace/github/aws/sagemaker-experiments/.tox/py38/lib/python3.8/site-packages/sklearn/utils/multiclass.py:14: DeprecationWarning: Please use `spmatrix` from the `scipy.sparse` namespace, the `scipy.sparse.base` namespace is deprecated.
    from scipy.sparse.base import spmatrix

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================================================================================ 189 passed, 4 skipped, 1 deselected, 1 warnings in 134.36 seconds =============================================================================================================================
py38 run-test: commands[1] | coverage report --fail-under=95
Name                                                                       Stmts   Miss  Cover
----------------------------------------------------------------------------------------------
.tox/py38/lib/python3.8/site-packages/smexperiments/__init__.py                2      0   100%
.tox/py38/lib/python3.8/site-packages/smexperiments/_base_types.py            95      7    93%
.tox/py38/lib/python3.8/site-packages/smexperiments/_boto_functions.py        34      0   100%
.tox/py38/lib/python3.8/site-packages/smexperiments/_environment.py           36      1    97%
.tox/py38/lib/python3.8/site-packages/smexperiments/_utils.py                 41      2    95%
.tox/py38/lib/python3.8/site-packages/smexperiments/api_types.py             149      4    97%
.tox/py38/lib/python3.8/site-packages/smexperiments/experiment.py             57      0   100%
.tox/py38/lib/python3.8/site-packages/smexperiments/metrics.py                79      4    95%
.tox/py38/lib/python3.8/site-packages/smexperiments/search_expression.py      47      0   100%
.tox/py38/lib/python3.8/site-packages/smexperiments/tracker.py               291     12    96%
.tox/py38/lib/python3.8/site-packages/smexperiments/training_job.py            6      0   100%
.tox/py38/lib/python3.8/site-packages/smexperiments/trial.py                  75      0   100%
.tox/py38/lib/python3.8/site-packages/smexperiments/trial_component.py        63      1    98%
----------------------------------------------------------------------------------------------
TOTAL                                                                        975     31    97%
_________________________________________________________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________________________________________________________
  py38: commands succeeded
  congratulations :)

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

Tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

danabens commented 1 year ago

checks failed, looks like github actions dropped support for 3.6:

Error: Version 3.6 with arch x64 not found
Available versions:

3.10.8 (x64)
3.11.0 (x64)
3.7.15 (x64)
3.8.14 (x64)
3.9.15 (x64)