The starter code stops with an error at the second step (src/eda)

fantauzzi commented 2 years ago

Steps to reproduce:

fork and clone the repository as per instructions in its README or in Udacity's course
still following those instructions, run the second step in the pipeline, which is already implemented: mlflow run src/eda the execution stops with an error, see below. (Note: the "Solving environment" took around 8 minute to complete on my system, with Intel i7-10700K and 32 GB of RAM)

2022/09/07 15:53:08 INFO mlflow.utils.conda: === Creating conda environment mlflow-8af91d08d13566aada991fdb38ca18c4510876b2 ===
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
imagehash-4.3.0      | 294 KB    | ################################################################################################################################################################################# | 100% 
widgetsnbextension-4 | 1.6 MB    | ################################################################################################################################################################################# | 100% 
tqdm-4.64.1          | 82 KB     | ################################################################################################################################################################################# | 100% 
jupyterlab_widgets-3 | 222 KB    | ################################################################################################################################################################################# | 100% 
ipywidgets-8.0.2     | 109 KB    | ################################################################################################################################################################################# | 100% 
ipython-8.5.0        | 552 KB    | ################################################################################################################################################################################# | 100% 
psutil-5.9.2         | 345 KB    | ################################################################################################################################################################################# | 100% 
prompt-toolkit-3.0.3 | 254 KB    | ################################################################################################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt']
Pip subprocess output:
Collecting wandb==0.10.31
  Using cached wandb-0.10.31-py2.py3-none-any.whl (1.8 MB)
Requirement already satisfied: six>=1.13.0 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (1.16.0)
Requirement already satisfied: PyYAML in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (6.0)
Requirement already satisfied: psutil>=5.0.0 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (5.9.2)
Requirement already satisfied: requests<3,>=2.0.0 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (2.28.1)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (2.8.2)
Collecting Click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting configparser>=3.8.1
  Using cached configparser-5.3.0-py3-none-any.whl (19 kB)
Collecting docker-pycreds>=0.4.0
  Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting GitPython>=1.0.0
  Using cached GitPython-3.1.27-py3-none-any.whl (181 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.9-py3-none-any.whl (63 kB)
Collecting promise<3,>=2.0
  Using cached promise-2.3-py3-none-any.whl
Collecting protobuf>=3.12.0
  Using cached protobuf-4.21.5-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (1.26.11)
Requirement already satisfied: idna<4,>=2.5 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (2022.6.15)
Requirement already satisfied: charset-normalizer<3,>=2 in /home/fanta/.local/miniconda3/envs/mlflow-8af91d08d13566aada991fdb38ca18c4510876b2/lib/python3.9/site-packages (from requests<3,>=2.0.0->wandb==0.10.31->-r /home/fanta/workspace/ml-pipeline-for-short-term-rental-prices/src/eda/condaenv.jnscdnqw.requirements.txt (line 1)) (2.1.1)
Collecting sentry-sdk>=0.4.0
  Using cached sentry_sdk-1.9.8-py2.py3-none-any.whl (158 kB)
Collecting shortuuid>=0.5.0
  Using cached shortuuid-1.0.9-py3-none-any.whl (9.4 kB)
Collecting smmap<6,>=3.0.1
  Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting subprocess32>=3.5.3
  Using cached subprocess32-3.5.4-py3-none-any.whl
Collecting pathtools
  Using cached pathtools-0.1.2-py3-none-any.whl
Installing collected packages: smmap, gitdb, subprocess32, shortuuid, sentry-sdk, protobuf, promise, pathtools, GitPython, docker-pycreds, configparser, Click, wandb
Successfully installed Click-8.1.3 GitPython-3.1.27 configparser-5.3.0 docker-pycreds-0.4.0 gitdb-4.0.9 pathtools-0.1.2 promise-2.3 protobuf-4.21.5 sentry-sdk-1.9.8 shortuuid-1.0.9 smmap-5.0.0 subprocess32-3.5.4 wandb-0.10.31
                                                                                                                                                                                                                          done
#
# To activate this environment, use
#
#     $ conda activate mlflow-8af91d08d13566aada991fdb38ca18c4510876b2
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... done
2022/09/07 16:01:24 INFO mlflow.projects.utils: === Created directory /tmp/tmp54_krbdj for downloading remote URIs passed to arguments of type 'path' ===
2022/09/07 16:01:24 INFO mlflow.projects.backend.local: === Running command 'source /home/fanta/.local/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-8af91d08d13566aada991fdb38ca18c4510876b2 1>&2 && jupyter notebook' in run with ID 'b9cd564571d340c3be96c73fa9513e08' === 
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [--debug] [subcommand]

Jupyter: Interactive Computing

positional arguments:
  subcommand     the subcommand to launch

optional arguments:
  -h, --help     show this help message and exit
  --version      show the versions of core jupyter packages and exit
  --config-dir   show Jupyter config dir
  --data-dir     show Jupyter data dir
  --runtime-dir  show Jupyter runtime dir
  --paths        show all Jupyter paths. Add --json for machine-readable format.
  --json         output paths as machine-readable json
  --debug        output debug information about paths

Available subcommands: bundlerextension dejavu execute kernel kernelspec lab labextension labhub migrate nbclassic nbconvert nbextension run server serverextension troubleshoot trust

Jupyter command `jupyter-notebook` not found.
2022/09/07 16:01:25 ERROR mlflow.cli: === Run (ID 'b9cd564571d340c3be96c73fa9513e08') failed ===

fantauzzi commented 2 years ago

I have been able to fix the problem by indicating jupyter instead of jupyterlab in the dependencies of src/eda/conda.yml. This is the content of my src/eda/conda.yml that fixed the problem

name: eda
channels:
  - conda-forge
  - defaults
dependencies:
  - jupyter=1.0.0
  - seaborn=0.11.2
  - pandas=1.4.4
  - pip=22.1.2
  - pandas-profiling=3.2.0
  - pyarrow=8.0.0
  - pip:
      - wandb==0.13.2

ChristophSchmidl commented 1 year ago

I had the same issue but just replaced "jupyter notebook" with "jupyter lab" in the MLproject file:

name: eda
conda_env: conda.yml

entry_points:
  main:
    command: jupyter lab

udacity / build-ml-pipeline-for-short-term-rental-prices

The starter code stops with an error at the second step (src/eda) #11