search

udacity / cd0581-building-a-reproducible-model-workflow-exercises

Exercise Staters and solutions for cd0581-building-a-reproducible-model-workflow by Giacomo Vianello
Other
20 stars 86 forks source link

exercise_10/solution stops with an error #7

Open fantauzzi opened 2 years ago

fantauzzi commented 2 years ago 

> mlflow run .
2022/09/04 11:18:25 INFO mlflow.utils.conda: === Creating conda environment mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6 ===
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
greenlet-1.1.3       | 91 KB     | ################################################################################################################################################################################# | 100% 
click-8.1.3          | 146 KB    | ################################################################################################################################################################################# | 100% 
protobuf-4.21.5      | 342 KB    | ################################################################################################################################################################################# | 100% 
omegaconf-2.0.6      | 103 KB    | ################################################################################################################################################################################# | 100% 
gunicorn-20.1.0      | 119 KB    | ################################################################################################################################################################################# | 100% 
sqlalchemy-1.4.40    | 2.4 MB    | ################################################################################################################################################################################# | 100% 
mlflow-1.14.1        | 4.2 MB    | ################################################################################################################################################################################# | 100% 
chardet-3.0.4        | 170 KB    | ################################################################################################################################################################################# | 100% 
docker-py-5.0.3      | 188 KB    | ################################################################################################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt']
Pip subprocess output:
Collecting wandb==0.10.21
  Using cached wandb-0.10.21-py2.py3-none-any.whl (2.0 MB)
Requirement already satisfied: GitPython>=1.0.0 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (3.1.27)
Requirement already satisfied: Click>=7.0 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (8.1.3)
Requirement already satisfied: docker-pycreds>=0.4.0 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (0.4.0)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (2.8.2)
Requirement already satisfied: configparser>=3.8.1 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (5.3.0)
Requirement already satisfied: requests<3,>=2.0.0 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: six>=1.13.0 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (1.16.0)
Requirement already satisfied: protobuf>=3.12.0 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (4.21.5)
Requirement already satisfied: PyYAML in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (6.0)
Requirement already satisfied: gitdb<5,>=4.0.1 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from GitPython>=1.0.0->wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (4.0.9)
Requirement already satisfied: smmap<6,>=3.0.1 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (3.0.5)
Collecting promise<3,>=2.0
  Using cached promise-2.3-py3-none-any.whl
Collecting psutil>=5.0.0
  Using cached psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (2022.6.15)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/condaenv.o2q2aa56.requirements.txt (line 1)) (1.25.11)
Collecting sentry-sdk>=0.4.0
  Using cached sentry_sdk-1.9.7-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.6-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.5-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.4-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.3-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.2-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.1-py2.py3-none-any.whl (157 kB)
  Using cached sentry_sdk-1.9.0-py2.py3-none-any.whl (156 kB)
Collecting shortuuid>=0.5.0
  Using cached shortuuid-1.0.9-py3-none-any.whl (9.4 kB)
Collecting subprocess32>=3.5.3
  Using cached subprocess32-3.5.4-py3-none-any.whl
Collecting pathtools
  Using cached pathtools-0.1.2-py3-none-any.whl
Installing collected packages: subprocess32, shortuuid, sentry-sdk, psutil, promise, pathtools, wandb
Successfully installed pathtools-0.1.2 promise-2.3 psutil-5.9.1 sentry-sdk-1.9.0 shortuuid-1.0.9 subprocess32-3.5.4 wandb-0.10.21
                                                                                                                                                                                                                          done
#
# To activate this environment, use
#
#     $ conda activate mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... done
2022/09/04 11:19:53 INFO mlflow.projects.utils: === Created directory /tmp/tmpfn3ljdp3 for downloading remote URIs passed to arguments of type 'path' ===
2022/09/04 11:19:53 INFO mlflow.projects.backend.local: === Running command 'source /home/fanta/.local/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6 1>&2 && python main.py $(echo '')' in run with ID '4dac944151114645ab75bbfaee227015' === 
2022/09/04 11:19:54 INFO mlflow.utils.conda: === Creating conda environment mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f ===
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
plotly-4.4.1         | 4.1 MB    | ################################################################################################################################################################################# | 100% 
retrying-1.3.3       | 11 KB     | ################################################################################################################################################################################# | 100% 
python-3.8.13        | 25.2 MB   | ################################################################################################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: - Ran pip subprocess with arguments:
['/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/random_forest/condaenv.lj2wi_mu.requirements.txt']
Pip subprocess output:
Collecting wandb==0.10.21
  Using cached wandb-0.10.21-py2.py3-none-any.whl (2.0 MB)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/random_forest/condaenv.lj2wi_mu.requirements.txt (line 1)) (2.8.2)
Requirement already satisfied: six>=1.13.0 in /home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages (from wandb==0.10.21->-r /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/random_forest/condaenv.lj2wi_mu.requirements.txt (line 1)) (1.16.0)
Collecting Click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting configparser>=3.8.1
  Using cached configparser-5.3.0-py3-none-any.whl (19 kB)
Collecting docker-pycreds>=0.4.0
  Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting GitPython>=1.0.0
  Using cached GitPython-3.1.27-py3-none-any.whl (181 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.9-py3-none-any.whl (63 kB)
Collecting promise<3,>=2.0
  Using cached promise-2.3-py3-none-any.whl
Collecting protobuf>=3.12.0
  Using cached protobuf-4.21.5-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
Collecting psutil>=5.0.0
  Using cached psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (284 kB)
Collecting requests<3,>=2.0.0
  Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.6.15-py3-none-any.whl (160 kB)
Collecting charset-normalizer<3,>=2
  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting sentry-sdk>=0.4.0
  Using cached sentry_sdk-1.9.7-py2.py3-none-any.whl (157 kB)
Collecting shortuuid>=0.5.0
  Using cached shortuuid-1.0.9-py3-none-any.whl (9.4 kB)
Collecting smmap<6,>=3.0.1
  Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting subprocess32>=3.5.3
  Using cached subprocess32-3.5.4-py3-none-any.whl
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting pathtools
  Using cached pathtools-0.1.2-py3-none-any.whl
Collecting PyYAML
  Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Installing collected packages: smmap, urllib3, idna, gitdb, charset-normalizer, certifi, subprocess32, shortuuid, sentry-sdk, requests, PyYAML, psutil, protobuf, promise, pathtools, GitPython, docker-pycreds, configparser, Click, wandb
Successfully installed Click-8.1.3 GitPython-3.1.27 PyYAML-6.0 certifi-2022.6.15 charset-normalizer-2.1.1 configparser-5.3.0 docker-pycreds-0.4.0 gitdb-4.0.9 idna-3.3 pathtools-0.1.2 promise-2.3 protobuf-4.21.5 psutil-5.9.1 requests-2.28.1 sentry-sdk-1.9.7 shortuuid-1.0.9 smmap-5.0.0 subprocess32-3.5.4 urllib3-1.26.12 wandb-0.10.21
                                                                                                                                                                                                                          done
#
# To activate this environment, use
#
#     $ conda activate mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... done
2022/09/04 11:20:56 INFO mlflow.projects.utils: === Created directory /tmp/tmpq_hnq_82 for downloading remote URIs passed to arguments of type 'path' ===
2022/09/04 11:20:56 INFO mlflow.projects.backend.local: === Running command 'source /home/fanta/.local/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f 1>&2 && python run.py --train_data exercise_6/data_train.csv:latest \
              --model_config /home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/outputs/2022-09-04/11-19-54/random_forest_config.json' in run with ID '1a76630e6bbc434f9cf6a81a83f227e0' === 
Traceback (most recent call last):
  File "run.py", line 15, in <module>
    import wandb
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/__init__.py", line 38, in <module>
    from wandb import sdk as wandb_sdk
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/sdk/__init__.py", line 12, in <module>
    from .wandb_init import init  # noqa: F401
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 29, in <module>
    from .backend.backend import Backend
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/sdk/backend/backend.py", line 17, in <module>
    from ..interface import interface
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 18, in <module>
    from wandb.proto import wandb_internal_pb2 as pb
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/proto/wandb_internal_pb2.py", line 15, in <module>
    from wandb.proto import wandb_telemetry_pb2 as wandb_dot_proto_dot_wandb__telemetry__pb2
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/wandb/proto/wandb_telemetry_pb2.py", line 34, in <module>
    _descriptor.FieldDescriptor(
  File "/home/fanta/.local/miniconda3/envs/mlflow-04dc18e87776aad288f8cabb3fd2dc65eef7059f/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
Traceback (most recent call last):
  File "/home/fanta/workspace/cd0581-building-a-reproducible-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_10/solution/main.py", line 26, in go
    _ = mlflow.run(
  File "/home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages/mlflow/projects/__init__.py", line 307, in run
    _wait_for(submitted_run_obj)
  File "/home/fanta/.local/miniconda3/envs/mlflow-35c2fef811a8212520e1943fc147f38ec6f203c6/lib/python3.9/site-packages/mlflow/projects/__init__.py", line 324, in _wait_for
    raise ExecutionException("Run (ID '%s') failed" % run_id)
mlflow.exceptions.ExecutionException: Run (ID '1a76630e6bbc434f9cf6a81a83f227e0') failed

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2022/09/04 11:20:57 ERROR mlflow.cli: === Run (ID '4dac944151114645ab75bbfaee227015') failed ===
fantauzzi commented 2 years ago

I have resolved it by replacing the content of conda.yml with the following

name: exercise_10
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - requests=2.24.0
  - pip=20.3.3
  - mlflow=1.28.0
  - hydra-core=1.0.6
  - pip:
      - wandb==0.13.2

and the content of random_forest/conda.yml with the following:

name: random_forest
channels:
  - conda-forge
  - defaults
dependencies:
  - pandas=1.4.4
  - pip=20.3.3
  - scikit-learn=0.24.1
  - matplotlib=3.5.3
  - plotly=5.10.0
  - pillow=9.2.0
  - pip:
      - wandb==0.13.2

Note that the code uses an obsolete version of scikit-learn.