search

neptune-ai / neptune-optuna

🚀 Optuna visualization dashboard that lets you log and monitor hyperparameter sweep live.
https://docs.neptune.ai/integrations/optuna/
Apache License 2.0
11 stars 5 forks source link

BUG: neptune and neptune-clients both installed, then it fails #42

Closed thistlillo closed 1 day ago

thistlillo commented 4 months ago

Describe the bug

I would like to log on neptune the results of a hyperparameter sweep using optuna. I follow the instructions on: https://docs.neptune.ai/integrations/optuna/

I tried several installation sequences, they all lead to failure.

Reproduction

conda install neptune neptune-optuna then follow the examples on the Neptune website

Expected behavior

Not such an error.

Traceback


python3 01_script.py\
        --data=./out/data.pkl \
        --epochs=20 \
        --train_size=0.7 \
        --test_size=0.1 \
        --out_fld=./out \
        --seed=10 \
        --shuffle_seed=20 \
        --data_seed=30 \
        --exp_name=“name” \
        --exp_description=“descr” \
        --lr=[1e-5, 1e-3] \
        --wd=[1-e-3, 1e-1] \
        --optim_trials=10 \
        --dev
Traceback (most recent call last):
  File "/mnt/home_new/user/workspace/project/project-wp1-nnn/01_script.py", line 5, in <module>
    import neptune
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/__init__.py", line 108, in <module>
    from neptune.metadata_containers import (
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/metadata_containers/__init__.py", line 24, in <module>
    from neptune.metadata_containers.metadata_container import MetadataContainer
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/metadata_containers/metadata_container.py", line 63, in <module>
    from neptune.internal.backends.factory import get_backend
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/internal/backends/factory.py", line 23, in <module>
    from .hosted_neptune_backend import HostedNeptuneBackend
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 42, in <module>
    from neptune.api.searching_entries import iter_over_pages
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/api/searching_entries.py", line 42, in <module>
    from neptune.internal.backends.hosted_client import DEFAULT_REQUEST_KWARGS
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/internal/backends/hosted_client.py", line 51, in <module>
    from neptune.version import version as neptune_client_version
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/version.py", line 63, in <module>
    __version__ = detect_version()
  File "/home/user/miniconda3/envs/ai/lib/python3.9/site-packages/neptune/version.py", line 46, in detect_version
    raise RuntimeError(
RuntimeError: We've detected that the 'neptune' and 'neptune-client' packages are both installed. Uninstall each of them and then install only the new 'neptune' package. For more information, see https://docs.neptune.ai/setup/upgrading/

Environment

This is the output of conda env export

name: ai
channels:
  - pytorch
  - huggingface
  - nvidia
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_kmp_llvm
  - abseil-cpp=20211102.0=h27087fc_1
  - absl-py=2.1.0=pyhd8ed1ab_0
  - accelerate=0.30.0=pyhd8ed1ab_0
  - aiohttp=3.9.5=py39hd1e30aa_0
  - aiosignal=1.3.1=pyhd8ed1ab_0
  - alembic=1.13.1=pyhd8ed1ab_1
  - arrow=1.2.3=py39h06a4308_1
  - arrow-cpp=14.0.2=h374c478_1
  - asttokens=2.0.5=pyhd3eb1b0_0
  - async-timeout=4.0.3=pyhd8ed1ab_0
  - attrs=22.1.0=py39h06a4308_0
  - aws-c-auth=0.6.19=h5eee18b_0
  - aws-c-cal=0.5.20=hff2c3d7_3
  - aws-c-common=0.8.5=h166bdaf_0
  - aws-c-compression=0.2.16=hf5f93bc_0
  - aws-c-event-stream=0.2.15=h6a678d5_0
  - aws-c-http=0.6.25=h5eee18b_0
  - aws-c-io=0.13.10=h2666983_0
  - aws-c-mqtt=0.7.13=h5eee18b_0
  - aws-c-s3=0.1.51=hdbd6064_0
  - aws-c-sdkutils=0.1.6=hf5f93bc_1
  - aws-checksums=0.1.13=hf5f93bc_5
  - aws-crt-cpp=0.18.16=h6a678d5_0
  - aws-sdk-cpp=1.10.55=h721c034_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - basemap=1.3.6=py39h1176785_1
  - basemap-data=1.3.6=py39h06a4308_1
  - blas=1.0=mkl
  - blinker=1.6.2=py39h06a4308_0
  - boost-cpp=1.70.0=ha2d47e9_1
  - boto3=1.34.82=py39h06a4308_0
  - botocore=1.34.82=py39h06a4308_0
  - bottleneck=1.3.5=py39h7deecbd_0
  - bravado=11.0.3=pyhd8ed1ab_0
  - bravado-core=5.17.1=pyhd8ed1ab_0
  - brotli=1.0.9=h5eee18b_7
  - brotli-bin=1.0.9=h5eee18b_7
  - brotli-python=1.0.9=py39h6a678d5_7
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.19.0=h5eee18b_0
  - ca-certificates=2024.6.2=hbcca054_0
  - cached-property=1.5.2=py_0
  - cartopy=0.21.1=py39h1176785_0
  - certifi=2024.2.2=py39h06a4308_0
  - cffi=1.16.0=py39h5eee18b_1
  - charset-normalizer=2.0.4=pyhd3eb1b0_0
  - click=8.1.7=py39h06a4308_0
  - colorlog=6.8.2=py39hf3d152e_0
  - contourpy=1.0.5=py39hdb19cb5_0
  - cryptography=42.0.5=py39hdda0065_1
  - cuda-cudart=11.8.89=0
  - cuda-cupti=11.8.87=0
  - cuda-libraries=11.8.0=0
  - cuda-nvrtc=11.8.89=0
  - cuda-nvtx=11.8.86=0
  - cuda-runtime=11.8.0=0
  - cycler=0.11.0=pyhd3eb1b0_0
  - cyrus-sasl=2.1.28=h52b45da_1
  - daal4py=2023.1.1=py39h79cecc1_0
  - dal=2023.1.1=hdb19cb5_48679
  - dataclasses=0.8=pyhc8e2a94_3
  - datasets=2.19.1=py_0
  - dbus=1.13.18=hb2f20db_0
  - debugpy=1.6.7=py39h6a678d5_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - dill=0.3.8=pyhd8ed1ab_0
  - dunamai=1.17.0=py39h06a4308_0
  - eccodes=2.26.0=hc08acdf_0
  - executing=0.8.3=pyhd3eb1b0_0
  - expat=2.5.0=h6a678d5_0
  - ffmpeg=4.3=hf484d3e_0
  - filelock=3.13.1=py39h06a4308_0
  - fontconfig=2.14.1=h4c34cd2_2
  - fonttools=4.25.0=pyhd3eb1b0_0
  - fqdn=1.5.1=pyhd8ed1ab_0
  - freeglut=3.2.1=h9c3ff4c_2
  - freetype=2.12.1=h4a9f257_0
  - frozenlist=1.4.1=py39hd1e30aa_0
  - fsspec=2024.3.1=pyhca7485f_0
  - future=0.18.3=py39h06a4308_0
  - geos=3.8.0=he6710b0_0
  - gflags=2.2.2=he1b5a44_1004
  - giflib=5.2.1=h5eee18b_3
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.37=py39h06a4308_0
  - glib=2.69.1=he621ea3_2
  - glog=0.5.0=h48cff8f_0
  - gmp=6.2.1=h295c915_3
  - gmpy2=2.1.2=py39heeb90bb_0
  - gnutls=3.6.15=he1e5248_0
  - greenlet=1.1.2=py39h5a03fae_2
  - grpc-cpp=1.48.2=he1ff14a_1
  - grpcio=1.48.2=py39he1ff14a_1
  - gst-plugins-base=1.14.1=h6a678d5_1
  - gstreamer=1.14.1=h5eee18b_1
  - hdf4=4.2.13=h3ca952b_2
  - hdf5=1.12.1=h2b7332f_3
  - huggingface_hub=0.23.0=pyhd8ed1ab_0
  - humanize=3.10.0=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.4=py39h06a4308_0
  - importlib-metadata=6.0.0=py39h06a4308_0
  - importlib_metadata=6.0.0=hd3eb1b0_0
  - importlib_resources=6.4.0=pyhd8ed1ab_0
  - intel-openmp=2023.1.0=hdb19cb5_46305
  - ipykernel=6.15.0=pyh210e3f2_0
  - ipython=8.12.2=py39h06a4308_0
  - ipywidgets=8.0.4=py39h06a4308_0
  - isoduration=20.11.0=pyhd8ed1ab_0
  - jasper=2.0.33=ha77e612_0
  - jedi=0.18.1=py39h06a4308_1
  - jinja2=3.1.3=py39h06a4308_0
  - jmespath=1.0.1=py39h06a4308_0
  - joblib=1.2.0=py39h06a4308_0
  - jpeg=9e=h5eee18b_1
  - jsonpointer=2.1=pyhd3eb1b0_0
  - jsonref=1.1.0=pyhd8ed1ab_0
  - jsonschema=4.17.3=py39h06a4308_0
  - jsonschema-with-format=4.17.3=pyhd8ed1ab_0
  - jupyter_client=8.1.0=py39h06a4308_0
  - jupyter_core=5.3.0=py39h06a4308_0
  - jupyterlab_widgets=3.0.5=py39h06a4308_0
  - jupyterthemes=0.20.0=py_1
  - kiwisolver=1.4.4=py39h6a678d5_0
  - krb5=1.20.1=h143b758_1
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.38=h1181459_1
  - lerc=3.0=h295c915_0
  - lesscpy=0.15.1=pyhd8ed1ab_0
  - libaec=1.0.6=h9c3ff4c_0
  - libbrotlicommon=1.0.9=h5eee18b_7
  - libbrotlidec=1.0.9=h5eee18b_7
  - libbrotlienc=1.0.9=h5eee18b_7
  - libclang=14.0.6=default_hc6dbbc7_1
  - libclang13=14.0.6=default_he11475f_1
  - libcublas=11.11.3.6=0
  - libcufft=10.9.0.58=0
  - libcufile=1.9.1.3=0
  - libcups=2.4.2=h2d74bed_1
  - libcurand=10.3.5.147=0
  - libcurl=8.2.1=h251f7ec_0
  - libcusolver=11.4.1.48=0
  - libcusparse=11.7.5.86=0
  - libdeflate=1.17=h5eee18b_0
  - libedit=3.1.20221030=h5eee18b_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.12=hdbd6064_1
  - libffi=3.4.4=h6a678d5_0
  - libgcc-ng=13.1.0=he5830b7_0
  - libgfortran-ng=11.2.0=h00389a5_1
  - libgfortran5=11.2.0=h1234567_1
  - libglu=9.0.0=hf484d3e_1
  - libiconv=1.16=h7f8727e_2
  - libidn2=2.3.4=h5eee18b_0
  - libjpeg-turbo=2.0.0=h9bf148f_0
  - libllvm14=14.0.6=hdb19cb5_3
  - libnetcdf=4.8.1=h14805e7_4
  - libnghttp2=1.52.0=h2d74bed_1
  - libnpp=11.8.0.86=0
  - libnvjpeg=11.9.0.86=0
  - libpng=1.6.39=h5eee18b_0
  - libpq=12.15=hdbd6064_1
  - libprotobuf=3.20.3=he621ea3_0
  - libsodium=1.0.18=h7b6447c_0
  - libssh2=1.10.0=hdbd6064_2
  - libstdcxx-ng=11.2.0=h1234567_1
  - libtasn1=4.19.0=h5eee18b_0
  - libthrift=0.15.0=hcc01f38_0
  - libtiff=4.5.1=h6a678d5_0
  - libunistring=0.9.10=h27cfd23_0
  - libuuid=1.41.5=h5eee18b_0
  - libwebp=1.2.4=h11a3e52_1
  - libwebp-base=1.2.4=h5eee18b_1
  - libxcb=1.15=h7f8727e_0
  - libxkbcommon=1.0.1=h5eee18b_1
  - libxml2=2.10.4=hcbfbd50_0
  - libxslt=1.1.37=h2085143_0
  - libzip=1.8.0=h6ac8c49_1
  - llvm-openmp=14.0.6=h9e868ea_0
  - lz4-c=1.9.4=h6a678d5_0
  - mako=1.3.5=pyhd8ed1ab_0
  - markdown=3.6=pyhd8ed1ab_0
  - markupsafe=2.1.3=py39h5eee18b_0
  - matplotlib=3.6.2=py39h06a4308_0
  - matplotlib-base=3.6.2=py39h945d387_0
  - matplotlib-inline=0.1.6=py39h06a4308_0
  - mizani=0.9.2=py39h06a4308_0
  - mkl=2023.1.0=h213fc3f_46343
  - mkl-service=2.4.0=py39h5eee18b_1
  - mkl_fft=1.3.6=py39h417a72b_1
  - mkl_random=1.2.2=py39h417a72b_1
  - monotonic=1.5=py_0
  - mpc=1.1.0=h10f8cd9_1
  - mpfr=4.0.2=hb69a4c5_1
  - mpi=1.0=mpich
  - mpich=4.1.1=hbae89fd_0
  - mpmath=1.3.0=py39h06a4308_0
  - msgpack-python=1.0.3=py39hd09550d_0
  - multidict=6.0.5=py39hd1e30aa_0
  - multiprocess=0.70.16=py39hd1e30aa_0
  - munkres=1.1.4=py_0
  - mysql=5.7.24=h721c034_2
  - nbformat=5.7.0=py39h06a4308_0
  - ncurses=6.4=h6a678d5_0
  - neptune=1.10.4=pyhd8ed1ab_0
  - neptune-client=1.10.4=pyhd8ed1ab_0
  - neptune-optuna=1.2.0=pyhd8ed1ab_0
  - nest-asyncio=1.5.6=py39h06a4308_0
  - nettle=3.7.3=hbbd107a_1
  - networkx=3.1=py39h06a4308_0
  - nspr=4.35=h6a678d5_0
  - nss=3.89.1=h6a678d5_0
  - numexpr=2.8.4=py39hc78ab66_1
  - numpy=1.25.0=py39h5f9d8c6_0
  - numpy-base=1.25.0=py39hb5e798b_0
  - oauthlib=3.2.2=py39h06a4308_0
  - openh264=2.1.1=h4ff587b_0
  - openssl=3.3.0=h4ab18f5_3
  - optuna=3.6.0=pyhd8ed1ab_0
  - orc=1.7.4=hb3bc3d3_1
  - packaging=23.1=py39h06a4308_0
  - pandas=1.5.3=py39h417a72b_0
  - parso=0.8.3=pyhd3eb1b0_0
  - patsy=0.5.3=py39h06a4308_0
  - pcre=8.45=h295c915_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=9.4.0=py39h6a678d5_0
  - pip=23.2.1=py39h06a4308_0
  - platformdirs=3.10.0=py39h06a4308_0
  - plotly=5.19.0=py39h2f386ee_0
  - plotnine=0.12.1=py39h06a4308_0
  - ply=3.11=py39h06a4308_0
  - poetry-dynamic-versioning=0.23.0=py39h06a4308_0
  - proj=8.2.1=ha227179_0
  - prompt-toolkit=3.0.36=py39h06a4308_0
  - protobuf=3.20.3=py39h6a678d5_0
  - psutil=5.9.0=py39h5eee18b_0
  - ptyprocess=0.7.0=pyhd3eb1b0_2
  - pure_eval=0.2.2=pyhd3eb1b0_0
  - pyarrow=14.0.2=py39h1eedbd7_0
  - pyarrow-hotfix=0.6=pyhd8ed1ab_0
  - pycparser=2.21=pyhd3eb1b0_0
  - pygments=2.15.1=py39h06a4308_1
  - pygrib=2.1.4=py39h8f7b310_2
  - pyjwt=2.8.0=py39h06a4308_0
  - pyparsing=3.0.9=py39h06a4308_0
  - pyproj=3.4.1=py39hb83c990_0
  - pyqt=5.15.7=py39h6a678d5_1
  - pyqt5-sip=12.11.0=py39h6a678d5_1
  - pyrsistent=0.18.0=py39heee7806_0
  - pyshp=2.1.3=pyhd3eb1b0_0
  - pysocks=1.7.1=py39h06a4308_0
  - python=3.9.16=h955ad1f_3
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-fastjsonschema=2.16.2=py39h06a4308_0
  - python-xxhash=3.4.1=py39hd1e30aa_0
  - python_abi=3.9=2_cp39
  - pytorch=2.2.2=py3.9_cuda11.8_cudnn8.7.0_0
  - pytorch-cuda=11.8=h7e8668a_5
  - pytorch-mutex=1.0=cuda
  - pytz=2022.7=py39h06a4308_0
  - pyyaml=6.0.1=py39h5eee18b_0
  - pyzmq=25.1.0=py39h6a678d5_0
  - qt-main=5.15.2=h7358343_9
  - qt-webengine=5.15.9=h9ab4d14_7
  - qtwebkit=5.212=h3fafdc1_5
  - re2=2022.04.01=h27087fc_0
  - readline=8.2=h5eee18b_0
  - regex=2023.10.3=py39h5eee18b_0
  - requests=2.31.0=py39h06a4308_1
  - requests-oauthlib=1.3.0=py_0
  - rfc3339-validator=0.1.4=py39h06a4308_0
  - rfc3987=1.3.8=py_0
  - s2n=1.3.27=h1b5e8cb_0
  - s3transfer=0.10.1=py39h06a4308_0
  - safetensors=0.4.3=py39h9fdd4d6_0
  - scikit-learn=1.2.2=py39h6a678d5_1
  - scikit-learn-intelex=2023.1.1=py39h06a4308_0
  - scipy=1.11.1=py39h5f9d8c6_0
  - seaborn=0.13.2=hd8ed1ab_0
  - seaborn-base=0.13.2=pyhd8ed1ab_0
  - seqeval=1.2.2=pyhd3deb0d_0
  - setuptools=68.0.0=py39h06a4308_0
  - shapely=2.0.1=py39h0fbb895_0
  - simplejson=3.17.6=py39h7f8727e_0
  - sip=6.6.2=py39h6a678d5_0
  - six=1.16.0=pyhd3eb1b0_1
  - smmap=4.0.0=pyhd3eb1b0_0
  - snappy=1.1.9=hbd366e4_1
  - sqlalchemy=2.0.30=py39hd3abc70_0
  - sqlite=3.41.2=h5eee18b_0
  - stack_data=0.2.0=pyhd3eb1b0_0
  - statsmodels=0.14.0=py39ha9d4c09_0
  - swagger-spec-validator=3.0.3=pyhd8ed1ab_0
  - sympy=1.12=py39h06a4308_0
  - tabulate=0.9.0=py39h06a4308_0
  - tbb=2021.8.0=hdb19cb5_0
  - tenacity=8.2.2=py39h06a4308_0
  - tensorboard=2.16.2=pyhd8ed1ab_0
  - tensorboard-data-server=0.7.0=py39hd4f0224_1
  - threadpoolctl=2.2.0=pyh0d69192_0
  - tk=8.6.12=h1ccaba5_0
  - tokenizers=0.15.1=py39h22610ee_0
  - toml=0.10.2=pyhd3eb1b0_0
  - tomlkit=0.11.1=py39h06a4308_0
  - torchaudio=2.2.2=py39_cu118
  - torchinfo=1.8.0=pyhd8ed1ab_0
  - torchtriton=2.2.0=py39
  - torchvision=0.17.2=py39_cu118
  - tornado=6.3.2=py39h5eee18b_0
  - tqdm=4.65.0=py39hb070fc8_0
  - traitlets=5.7.1=py39h06a4308_0
  - transformers=4.39.3=pyhd8ed1ab_0
  - typing=3.10.0.0=py39h06a4308_0
  - typing-extensions=4.7.1=py39h06a4308_0
  - typing_extensions=4.7.1=py39h06a4308_0
  - tzdata=2023c=h04d1e81_0
  - uri-template=1.3.0=pyhd8ed1ab_0
  - urllib3=1.26.18=pyhd8ed1ab_0
  - utf8proc=2.6.1=h5eee18b_1
  - wcwidth=0.2.5=pyhd3eb1b0_0
  - webcolors=1.13=pyhd8ed1ab_0
  - websocket-client=1.8.0=py39h06a4308_0
  - werkzeug=3.0.2=pyhd8ed1ab_0
  - wheel=0.38.4=py39h06a4308_0
  - widgetsnbextension=4.0.5=py39h06a4308_0
  - xorg-fixesproto=5.0=h7f98852_1002
  - xorg-inputproto=2.3.2=h7f98852_1002
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libx11=1.8.6=h8ee46fc_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxfixes=5.0.3=h7f98852_1004
  - xorg-libxi=1.7.10=h7f98852_0
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xproto=7.0.31=h27cfd23_1007
  - xxhash=0.8.2=hd590300_0
  - xz=5.4.2=h5eee18b_0
  - yaml=0.2.5=h7b6447c_0
  - yarl=1.9.4=py39hd1e30aa_0
  - zeromq=4.3.4=h2531618_0
  - zipp=3.11.0=py39h06a4308_0
  - zlib=1.2.13=h5eee18b_0
  - zstd=1.5.5=hc292b87_0
  - pip:
      - evaluate==0.4.2
      - llvmlite==0.42.0
      - nervaluate==0.1.8
      - numba==0.59.1
      - pynndescent==0.5.12
      - umap-learn==0.5.6
prefix: /home/user/miniconda3/envs/ai

linux

Python 3.9.16

SiddhantSadangi commented 4 months ago

Hey @thistlillo Franco 👋 Thank you for bringing this to our attention! ​ I have created a ticket for the dev team to update the conda recipe for neptune-optuna, and I will update this thread once it is done ✅ ​ Meanwhile, is it possible for you to use pip to install packages? If yes, you can uninstall the conda installations of neptune, neptune-client, and neptune-optuna by running the below: ​conda uninstall neptune neptune-client neptune-optuna -y ​ and then just install neptune and neptune-optuna using pip as below: ​pip install -U neptune[optuna] ​ Please let me know if this works for you.

SiddhantSadangi commented 1 day ago

We are not distributing on conda anymore.

Would request you to install via pip.

thistlillo commented 1 day ago

Ok, thank you. That is what I do, but it leads to messy situation.

Please notice that on remote servers your users do not usually have sudo access and it is almost impossible to work without a simple way to change, for example, the current python version available on those systems. In these cases, pip does not help.

SiddhantSadangi commented 1 day ago

Pip should take the python version into account when installing packages. Is that not the case with your setup? 🤔

thistlillo commented 1 day ago

No, I am saying that sometimes I need a specific Python version for my ML pipeline, which I can install through conda. That's what I have been doing so far. Installing/uninstalling libraries in the same conda env using both conda and pip has led (neither so often nor once or twice) to a complete mess.