ReadTimeout during hyperparameter tuning

[ZiyueWang25]

Bug description

Some trials finished in error state and the log is like this: error_eg.txt

Steps to reproduce

git checkout benchmark-pr

python src/imitation/scripts/tuning.py with bc 'parallel_run_config.base_named_configs=["bc_seals_hopper"]'

Environment

• Operating system and version: Ubuntu 22.02

• Python version:

• Output of pip freeze --all:

  absl-py==2.0.0
  aiohttp==3.8.4
  aiosignal==1.3.1
  alabaster==0.7.13
  ale-py==0.8.1
  alembic==1.12.0
  altair==5.0.1
  annotated-types==0.5.0
  anyio==3.7.1
  appdirs==1.4.4
  apsw==3.36.0.post1
  apturl==0.5.2
  argon2-cffi==21.1.0
  asciinema==2.1.0
  asciitree==0.3.3
  async-timeout==4.0.2
  attrs==22.2.0
  autopep8==1.6.0
  AutoROM==0.4.2
  AutoROM.accept-rom-license==0.6.1
  Babel==2.8.0
  backcall==0.2.0
  bcrypt==3.2.0
  beautifulsoup4==4.10.0
  beniget==0.4.1
  binance==0.3
  black==22.6.0
  bleach==4.1.0
  blinker==1.4
  Brlapi==0.8.3
  Brotli==1.0.9
  cachetools==5.3.1
  caffeine==2.9.8
  certifi==2020.6.20
  cffi==1.15.1
  cfgv==3.4.0
  chardet==4.0.0
  charset-normalizer==3.2.0
  click==8.0.3
  cloudpickle==2.2.1
  cmaes==0.10.0
  cmake==3.26.4
  codecov==2.1.13
  codespell==2.1.0
  -e git+https://github.com/openai/coinrun.git@523704f3a203dcaad84caf96ea92799452dc902f#egg=coinrun
  colorama==0.4.4
  colorlog==6.7.0
  comm==0.1.4
  command-not-found==0.3
  coverage==6.4.4
  cryptography==3.4.8
  css-parser==1.0.7
  cssselect==1.1.0
  cssutils==1.0.2
  cupshelpers==1.0
  cycler==0.11.0
  darglint==1.8.1
  dataclasses-json==0.5.14
  datasets==2.14.5
  dateparser==1.1.8
  dblatex==0.3.12
  dbus-python==1.2.18
  debugpy==1.8.0
  decorator==4.4.2
  defer==1.0.6
  defusedxml==0.7.1
  dill==0.3.7
  distlib==0.3.7
  distro==1.8.0
  distro-info==1.0
  dnspython==2.1.0
  docker==5.0.3
  docker-compose==1.29.2
  docker-pycreds==0.4.0
  dockerpty==0.4.1
  docopt==0.6.2
  docutils==0.17.1
  duplicity==0.8.21
  einops==0.6.1
  entrypoints==0.4
  ewmh==0.1.6
  exceptiongroup==1.1.3
  execnet==2.0.2
  Farama-Notifications==0.0.4
  fasteners==0.14.1
  fastjsonschema==2.18.0
  feedparser==6.0.8
  filelock==3.7.1
  flake8==4.0.1
  flake8-blind-except==0.2.1
  flake8-builtins==1.5.3
  flake8-commas==2.1.0
  flake8-debugger==4.1.2
  flake8-docstrings==1.6.0
  flake8-isort==4.1.2.post0
  fonttools==4.29.1
  frozenlist==1.3.3
  fs==2.4.12
  fsspec==2023.6.0
  furo==2022.6.21
  future==0.18.2
  gast==0.5.2
  gattlib==0.20201113
  gitdb==4.0.10
  GitPython==3.1.31
  glcontext==2.3.7
  glfw==1.12.0
  google-auth==2.23.2
  google-auth-oauthlib==0.4.6
  greenlet==2.0.2
  grpcio==1.43.0
  gym==0.21.0
  gym-notices==0.0.8
  gym3==0.3.3
  gymnasium==0.29.1
  html2text==2020.1.16
  html5-parser==0.4.10
  html5lib==1.1
  httplib2==0.20.2
  huggingface-hub==0.17.3
  huggingface-sb3==3.0
  hypothesis==6.54.6
  identify==2.5.30
  idna==3.3
  ifaddr==0.1.7
  imageio==2.31.1
  imageio-ffmpeg==0.3.0
  imagesize==1.4.1
  imitation @ file:///home/vincentwang/far/imitation
  importlab==0.8
  importlib-metadata==4.13.0
  importlib-resources==6.1.0
  iniconfig==2.0.0
  ipdb==0.13.13
  ipykernel==6.15.3
  ipython==7.31.1
  ipython_genutils==0.2.0
  ipywidgets==8.1.1
  isort==5.12.0
  itchat==1.3.10
  jax-jumpy==1.0.0
  jaxtyping==0.2.20
  jedi==0.18.0
  jeepney==0.7.1
  Jinja2==3.1.2
  joblib==1.3.2
  jsonpickle==3.0.2
  jsonschema==3.2.0
  jupyter==1.0.0
  jupyter-cache==0.6.1
  jupyter-client==6.1.12
  jupyter-console==6.4.2
  jupyter-server==1.24.0
  jupyter-server-mathjax==0.2.6
  jupyter_core==5.3.2
  jupyterlab-pygments==0.1.2
  jupyterlab-widgets==3.0.9
  kazam==1.4.5
  keyring==23.5.0
  kiwisolver==1.3.2
  langchain==0.0.268
  langsmith==0.0.25
  language-selector==0.1
  launchpadlib==1.10.16
  lazr.restfulclient==0.14.4
  lazr.uri==1.0.6
  libcst==1.0.1
  lit==16.0.6
  livereload==2.6.3
  lockfile==0.12.2
  louis==3.20.0
  lxml==4.8.0
  lz4==3.1.3+dfsg
  macaroonbakery==1.3.1
  Mako==1.1.3
  Markdown==3.3.6
  markdown-it-py==2.2.0
  MarkupSafe==2.1.1
  marshmallow==3.20.1
  matplotlib==3.5.1
  matplotlib-inline==0.1.3
  mccabe==0.6.1
  mdit-py-plugins==0.3.5
  mdurl==0.1.2
  mechanize==0.4.7
  mistune==3.0.2
  moderngl==5.8.2
  monotonic==1.6
  more-itertools==8.10.0
  moviepy==1.0.3
  mpl-finance==0.10.1
  mplfinance==0.12.9b7
  mpmath==0.0.0
  msgpack==1.0.3
  mujoco==2.3.7
  multidict==6.0.4
  multiprocess==0.70.15
  munch==2.5.0
  mypy==0.991
  mypy-extensions==1.0.0
  myst-nb==0.17.2
  myst-parser==0.18.1
  nbclient==0.5.13
  nbconvert==7.8.0
  nbdime==3.2.1
  nbformat==5.9.2
  nest-asyncio==1.5.4
  netifaces==0.11.0
  networkx==2.8.3
  ninja==1.11.1
  nodeenv==1.8.0
  notebook==6.4.8
  numexpr==2.8.5
  numpy==1.25.2
  nvidia-cublas-cu11==11.10.3.66
  nvidia-cuda-cupti-cu11==11.7.101
  nvidia-cuda-nvrtc-cu11==11.7.99
  nvidia-cuda-runtime-cu11==11.7.99
  nvidia-cudnn-cu11==8.5.0.96
  nvidia-cufft-cu11==10.9.0.58
  nvidia-curand-cu11==10.2.10.91
  nvidia-cusolver-cu11==11.4.0.1
  nvidia-cusparse-cu11==11.7.4.91
  nvidia-nccl-cu11==2.14.3
  nvidia-nvtx-cu11==11.7.91
  oauthlib==3.2.0
  olefile==0.46
  openai==0.27.8
  opencv-python==4.8.1.78
  optuna==3.3.0
  packaging==21.3
  pandas==1.4.4
  pandocfilters==1.5.0
  paramiko==2.9.3
  parso==0.8.1
  pathspec==0.11.2
  pathtools==0.1.2
  pexpect==4.8.0
  pickleshare==0.7.5
  Pillow==9.0.1
  pip==23.2.1
  platformdirs==3.10.0
  plotly==5.4.0
  pluggy==1.3.0
  ply==3.11
  pockets==0.9.1
  pre-commit==3.4.0
  procgen==0.10.7
  proglog==0.1.10
  prometheus-client==0.9.0
  promise==2.3
  prompt-toolkit==3.0.28
  protobuf==3.20.3
  psutil==5.9.5
  ptyprocess==0.7.0
  py==1.10.0
  py-cpuinfo==9.0.0
  py7zr==0.11.3+dfsg
  pyarrow==12.0.1
  pyasn1==0.5.0
  pyasn1-modules==0.3.0
  PyBluez==0.23
  pycairo==1.20.1
  pychm==0.8.6
  pycnite==2023.9.18
  pycodestyle==2.8.0
  pycparser==2.21
  pycryptodome==3.18.0
  pycryptodomex==3.11.0
  pycups==2.0.1
  pydantic==1.10.13
  pydantic_core==2.6.1
  pydeck==0.8.1b0
  pydocstyle==6.3.0
  pydot==1.4.2
  pyenchant==3.2.0
  pyflakes==2.4.0
  pygame==2.1.3
  Pygments==2.15.1
  PyGObject==3.42.1
  pyinotify==0.9.6
  PyJWT==2.3.0
  pylibiio==0.23
  pymacaroons==0.13.0
  pymol==2.5.0
  Pympler==1.0.1
  PyNaCl==1.5.0
  PyOpenGL==3.1.5
  pyparsing==2.4.7
  pypng==0.20220715.0
  PyQRCode==1.2.1
  PyQt5==5.15.6
  PyQt5-sip==12.9.1
  PyQtWebEngine==5.15.5
  pyRFC3339==1.1
  pyrsistent==0.18.1
  pytest==7.1.3
  pytest-cov==3.0.0
  pytest-flakefinder==1.1.0
  pytest-forked==1.6.0
  pytest-randomly==3.15.0
  pytest-timeout==2.1.0
  pytest-xdist==2.5.0
  pytest_notebook==0.8.0
  python-apt==2.4.0+ubuntu1
  python-binance==1.0.17
  python-dateutil==2.8.2
  python-debian==0.1.43+ubuntu1.1
  python-dotenv==0.19.2
  python-Levenshtein==0.12.2
  python-xlib==0.29
  pythran==0.10.0
  pytype==2022.7.26
  pytz==2022.1
  pytz-deprecation-shim==0.1.0.post0
  pyxattr==0.7.2
  pyxdg==0.27
  PyYAML==6.0.1
  pyzmq==22.3.0
  qtconsole==5.4.4
  QtPy==2.4.0
  ray==2.0.1
  regex==2021.11.10
  reportlab==3.6.8
  repoze.lru==0.7
  requests==2.31.0
  requests-oauthlib==1.3.1
  requests-toolbelt==0.9.1
  rich==13.4.2
  Routes==2.5.1
  rsa==4.9
  ruamel.yaml==0.17.32
  ruamel.yaml.clib==0.2.7
  sacred==0.8.4
  scikit-learn==1.3.1
  scipy==1.9.3
  screen-resolution-extra==0.0.0
  seaborn==0.12.2
  seals==0.2.1
  SecretStorage==3.3.1
  Send2Trash==1.8.1b0
  sentry-sdk==1.26.0
  setproctitle==1.3.2
  setuptools==65.5.0
  setuptools-scm==7.0.5
  sgmllib3k==1.0.0
  shell_gpt==0.9.4
  Shimmy==0.2.1
  shortuuid==1.0.11
  six==1.16.0
  smmap==5.0.0
  sniffio==1.3.0
  snowballstemmer==2.2.0
  sortedcontainers==2.4.0
  soupsieve==2.3.1
  Sphinx==5.1.1
  sphinx-autobuild==2021.3.14
  sphinx-autodoc-typehints==1.19.2
  sphinx-basic-ng==1.0.0b2
  sphinx-copybutton==0.5.0
  sphinx-github-changelog==1.2.1
  sphinx-rtd-theme==1.0.0
  sphinxcontrib-applehelp==1.0.7
  sphinxcontrib-devhelp==1.0.5
  sphinxcontrib-htmlhelp==2.0.4
  sphinxcontrib-jsmath==1.0.1
  sphinxcontrib-napoleon==0.7
  sphinxcontrib-qthelp==1.0.6
  sphinxcontrib-serializinghtml==1.1.9
  SQLAlchemy==2.0.20
  ssh-import-id==5.11
  stable-baselines3==2.1.0
  streamlit==1.24.1
  sympy==1.9
  systemd-python==234
  tabulate==0.9.0
  tenacity==8.2.2
  tensorboard==2.11.2
  tensorboard-data-server==0.6.1
  tensorboard-plugin-wit==1.8.1
  tensorboardX==2.6.2.2
  terminado==0.13.1
  testpath==0.5.0
  texttable==1.6.4
  threadpoolctl==3.2.0
  tinycss2==1.2.1
  tokenize-rt==5.2.0
  toml==0.10.2
  tomli==2.0.1
  toolz==0.12.0
  torch==2.0.1
  torchaudio==2.0.2
  torchvision==0.15.2
  tornado==6.1
  tqdm==4.65.0
  traitlets==5.10.1
  triton==2.0.0
  typeguard==4.0.0
  typer==0.9.0
  typing-inspect==0.9.0
  typing_extensions==4.7.1
  tzdata==2023.3
  tzlocal==4.3.1
  ubuntu-advantage-tools==8001
  ubuntu-drivers-common==0.0.0
  ufoLib2==0.13.1
  ufw==0.36.1
  ujson==5.8.0
  ulauncher==5.15.2
  unattended-upgrades==0.1
  unicodedata2==14.0.0
  urllib3==2.0.3
  usb-creator==0.3.7
  validators==0.20.0
  vext==0.7.6
  virtualenv==20.21.1
  wadllib==1.3.6
  wandb==0.12.21
  wasabi==1.1.2
  watchdog==3.0.0
  wcwidth==0.2.5
  webencodings==0.5.1
  WebOb==1.8.6
  websocket-client==1.2.3
  websockets==11.0.3
  Werkzeug==3.0.0
  wheel==0.38.4
  widgetsnbextension==4.0.9
  wrapt==1.15.0
  xdg==5
  xkit==0.0.0
  xxhash==3.3.0
  yarl==1.9.2
  youtube-dl==2021.12.17
  zeroconf==0.38.3
  zipp==1.0.0

[ernestum]

Looks like a networking issue at the side of huggingface, where we download expert models and trajectories form.