search

mle-infrastructure / mle-toolbox

Lightweight Tool to Manage Distributed ML Experiments 🛠
https://mle-infrastructure.github.io/mle_toolbox/toolbox/
MIT License
2 stars 0 forks source link

`mle init` failed with `KeyError` in `load_mle_toolbox_config` #82

Closed denisalevi closed 2 years ago

denisalevi commented 2 years ago

Clean install:

conda create -n mle-toolbox python=3
pip install mle-toolbox
mle init

Got this error:

Traceback (most recent call last):
  File "/home/denis/.conda/envs/mle-toolbox/bin/mle", line 5, in <module>
    from mle_toolbox.toolbox import main
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/mle_toolbox/__init__.py", line 2, in <module>
    from .experiment import MLExperiment
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/mle_toolbox/experiment.py", line 1, in <module>
    from .utils.core_experiment import (
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/mle_toolbox/utils/__init__.py", line 1, in <module>
    from .core_experiment import (
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/mle_toolbox/utils/core_experiment.py", line 41, in <module>
    mle_config = load_mle_toolbox_config()
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/mle_toolbox/utils/core_files_load.py", line 38, in load_mle_toolbox_config
    if mle_config.general.use_credential_encryption:
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/dotmap/__init__.py", line 116, in __getattr__
    return self[k]
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/dotmap/__init__.py", line 93, in __getitem__
    return self._map[k]
KeyError: 'use_credential_encryption'

Using mle-toolbox 0.3.0.

Here are all packages in my conda environment (click on arrow)

``` # packages in environment at /home/denis/.conda/envs/mle-toolbox: # # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 4.5 1_gnu absl-py 0.13.0 pypi_0 pypi appdirs 1.4.4 py_0 astroid 2.6.6 py39h06a4308_0 attrs 21.2.0 pyhd3eb1b0_0 backcall 0.2.0 pyhd3eb1b0_0 bcrypt 3.2.0 pypi_0 pypi black 19.10b0 py_0 brotlipy 0.7.0 py39h27cfd23_1003 bzip2 1.0.8 h7b6447c_0 c-ares 1.17.1 h27cfd23_0 ca-certificates 2021.7.5 h06a4308_1 cachetools 4.2.2 pypi_0 pypi certifi 2021.5.30 py39h06a4308_0 cffi 1.14.6 py39h400218f_0 charset-normalizer 2.0.4 pyhd3eb1b0_0 click 8.0.1 pyhd3eb1b0_0 colorama 0.4.4 pypi_0 pypi commentjson 0.9.0 pypi_0 pypi commonmark 0.9.1 pypi_0 pypi conda 4.10.3 py39h06a4308_0 conda-package-handling 1.7.3 py39h27cfd23_1 cryptography 3.4.7 py39hd23ed53_0 cycler 0.10.0 pypi_0 pypi debugpy 1.4.1 py39h295c915_0 decorator 5.0.9 pyhd3eb1b0_0 dotmap 1.3.24 pypi_0 pypi flake8 3.9.2 pyhd3eb1b0_0 gitdb 4.0.7 pypi_0 pypi gitpython 3.1.18 pypi_0 pypi google-auth 1.35.0 pypi_0 pypi google-auth-oauthlib 0.4.5 pypi_0 pypi grpcio 1.39.0 pypi_0 pypi h5py 3.4.0 pypi_0 pypi icu 58.2 he6710b0_3 idna 3.2 pyhd3eb1b0_0 importlib-metadata 3.10.0 py39h06a4308_0 ipykernel 6.2.0 py39h06a4308_1 ipython 7.26.0 py39hb070fc8_0 ipython_genutils 0.2.0 pyhd3eb1b0_1 isort 5.9.3 pyhd3eb1b0_0 jedi 0.18.0 py39h06a4308_1 jupyter_client 6.1.12 pyhd3eb1b0_0 jupyter_core 4.7.1 py39h06a4308_0 kiwisolver 1.3.2 pypi_0 pypi krb5 1.19.2 hac12032_0 lark-parser 0.7.8 pypi_0 pypi lazy-object-proxy 1.6.0 py39h27cfd23_0 ld_impl_linux-64 2.35.1 h7274673_9 libarchive 3.4.2 h62408e4_0 libcurl 7.78.0 h0b77cf5_0 libedit 3.1.20210714 h7f8727e_0 libev 4.33 h7b6447c_0 libffi 3.3 he6710b0_2 libgcc-ng 9.3.0 h5101ec6_17 libgomp 9.3.0 h5101ec6_17 libnghttp2 1.41.0 hf8bcb03_2 libsodium 1.0.18 h7b6447c_0 libsolv 0.7.19 h780b84a_5 conda-forge libssh2 1.9.0 h1ba5d50_1 libstdcxx-ng 9.3.0 hd4cf53a_17 libxml2 2.9.12 h03d6c58_0 lz4-c 1.9.3 h295c915_1 mamba 0.7.6 py39heb6f48a_0 conda-forge markdown 3.3.4 pypi_0 pypi matplotlib 3.4.3 pypi_0 pypi matplotlib-inline 0.1.2 pyhd3eb1b0_2 mccabe 0.6.1 py39h06a4308_1 mle-logging 0.0.2 pypi_0 pypi mle-toolbox 0.3.0 pypi_0 pypi mypy_extensions 0.4.3 py39h06a4308_0 ncurses 6.2 he6710b0_1 numpy 1.21.2 pypi_0 pypi oauthlib 3.1.1 pypi_0 pypi openssl 1.1.1k h27cfd23_0 pandas 1.3.2 pypi_0 pypi paramiko 2.7.2 pypi_0 pypi parso 0.8.2 pyhd3eb1b0_0 pathspec 0.7.0 py_0 pexpect 4.8.0 pyhd3eb1b0_3 pickle5 0.0.11 pypi_0 pypi pickledb 0.9.2 pypi_0 pypi pickleshare 0.7.5 pyhd3eb1b0_1003 pillow 8.3.1 pypi_0 pypi pip 21.2.4 py37h06a4308_0 plotext 2.3.1 pypi_0 pypi prompt-toolkit 3.0.17 pyh06a4308_0 protobuf 3.17.3 pypi_0 pypi ptyprocess 0.7.0 pyhd3eb1b0_2 pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycodestyle 2.7.0 pyhd3eb1b0_0 pycosat 0.6.3 py39h27cfd23_0 pycparser 2.20 py_2 pyflakes 2.3.1 pyhd3eb1b0_0 pygments 2.10.0 pyhd3eb1b0_0 pylint 2.9.6 py39h06a4308_1 pynacl 1.4.0 pypi_0 pypi pyopenssl 20.0.1 pyhd3eb1b0_1 pyparsing 2.4.7 pypi_0 pypi pysocks 1.7.1 py39h06a4308_0 python 3.9.6 h12debd9_1 python-dateutil 2.8.2 pyhd3eb1b0_0 python_abi 3.9 2_cp39 conda-forge pytz 2021.1 pypi_0 pypi pyyaml 5.4.1 pypi_0 pypi pyzmq 22.2.1 py39h295c915_1 readline 8.1 h27cfd23_0 regex 2021.8.3 py39h7f8727e_0 reproc 14.2.1 h36c2ea0_0 conda-forge reproc-cpp 14.2.1 h58526e2_0 conda-forge requests 2.26.0 pyhd3eb1b0_0 requests-oauthlib 1.3.0 pypi_0 pypi rich 9.13.0 pypi_0 pypi rsa 4.7.2 pypi_0 pypi ruamel_yaml 0.15.100 py39h27cfd23_0 scipy 1.7.1 pypi_0 pypi scp 0.13.6 pypi_0 pypi seaborn 0.11.2 pypi_0 pypi setuptools 52.0.0 py39h06a4308_0 six 1.16.0 pyhd3eb1b0_0 smmap 4.0.0 pypi_0 pypi sqlite 3.36.0 hc218d9a_0 sshtunnel 0.4.0 pypi_0 pypi tensorboard 2.6.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.0 pypi_0 pypi tk 8.6.10 hbc83047_0 toml 0.10.2 pyhd3eb1b0_0 torch 1.9.0 pypi_0 pypi tornado 6.1 py39h27cfd23_0 tqdm 4.62.1 pyhd3eb1b0_1 traitlets 5.0.5 pyhd3eb1b0_0 typed-ast 1.4.3 py39h7f8727e_1 typing_extensions 3.10.0.0 pyh06a4308_0 tzdata 2021a h5d7bf9c_0 urllib3 1.26.6 pyhd3eb1b0_1 wcwidth 0.2.5 py_0 werkzeug 2.0.1 pypi_0 pypi wheel 0.37.0 pyhd3eb1b0_0 wrapt 1.12.1 py39he8ac12f_1 xz 5.2.5 h7b6447c_0 yaml 0.2.5 h7b6447c_0 zeromq 4.3.4 h2531618_0 zipp 3.5.0 pyhd3eb1b0_0 zlib 1.2.11 h7b6447c_3 zstd 1.4.9 haebb681_0 ```

RobertTLange commented 2 years ago

Hi Denis, sorry right now everything is still very rough and mle init is pretty much broken and has to be redesigned. Maybe I will get completely rid off it. Same holds for the encryption part. It doesn't really make a lot of sense without adding a lot of complexity. The key error above is related to the mle_config.toml file.

https://github.com/RobertTLange/mle-toolbox/blob/ab038b596c248164cc02f083613e9f369b2fd72f/config_template.toml#L18

Are you using the template from the most recent version and have deleted any previous config file? I plan on cleaning everything up mid October in my vacation. Thank you for all the broken links! Rob

denisalevi commented 2 years ago

Ah yes, had an old config file. Thanks. After removing it, I get

Could not load mle-toolbox configuration .toml from ~/mle_config.tomlProceed with minimal config used for testing.
Traceback (most recent call last):
  File "/home/denis/.conda/envs/mle-toolbox/bin/mle", line 8, in <module>
    sys.exit(main())
  File "/home/denis/.conda/envs/mle-toolbox/lib/python3.9/site-packages/mle_toolbox/toolbox.py", line 50, in main
    from .src.initialize import initialize
ModuleNotFoundError: No module named 'mle_toolbox.src'

But no worries, just wanted to report it. I just wanted to use the monitor and followed the instructions :). I'll set up the config file manually then.