search

mlcommons / storage

MLPerf™ Storage Benchmark Suite
https://mlcommons.org/en/groups/research-storage/
Apache License 2.0
61 stars 19 forks source link

Hydra Error when trying to run datagen #21

Closed kmannthey closed 1 year ago

kmannthey commented 1 year ago

Version: commit bd21330419efd6ee08cccc1bf09cbc465ffa7ba4 (HEAD, tag: v0.5-rc1)

The Benchmark seems to install ok. 

[kmannthey@pgh-echo-1-1 storage]$ pip3.10  install -r dlio_benchmark/requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Requirement already satisfied: absl-py==1.3.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 4)) (1.3.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 5)) (4.9.3)
Requirement already satisfied: astunparse==1.6.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 6)) (1.6.3)
Requirement already satisfied: cachetools==5.2.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 7)) (5.2.0)
Requirement already satisfied: certifi==2022.9.24 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 8)) (2022.9.24)
Requirement already satisfied: charset-normalizer==2.1.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 9)) (2.1.1)
Requirement already satisfied: flatbuffers==22.10.26 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 10)) (22.10.26)
Requirement already satisfied: gast==0.4.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 11)) (0.4.0)
Requirement already satisfied: google-auth==2.14.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 12)) (2.14.1)
Requirement already satisfied: google-auth-oauthlib==0.4.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 13)) (0.4.6)
Requirement already satisfied: google-pasta==0.2.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 14)) (0.2.0)
Requirement already satisfied: grpcio==1.51.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 15)) (1.51.0)
Requirement already satisfied: h5py==3.7.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 16)) (3.7.0)
Requirement already satisfied: hydra-core==1.2.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 17)) (1.2.0)
Requirement already satisfied: idna==3.4 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 18)) (3.4)
Requirement already satisfied: keras==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 19)) (2.11.0)
Requirement already satisfied: libclang==14.0.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 20)) (14.0.6)
Requirement already satisfied: Markdown==3.4.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 21)) (3.4.1)
Requirement already satisfied: MarkupSafe==2.1.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 22)) (2.1.1)
Requirement already satisfied: mpi4py==3.1.4 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 23)) (3.1.4)
Requirement already satisfied: numpy==1.23.5 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 24)) (1.23.5)
Requirement already satisfied: oauthlib==3.2.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 25)) (3.2.2)
Requirement already satisfied: omegaconf==2.2.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 26)) (2.2.3)
Requirement already satisfied: opt-einsum==3.3.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 27)) (3.3.0)
Requirement already satisfied: packaging==21.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 28)) (21.3)
Requirement already satisfied: pandas==1.5.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 29)) (1.5.1)
Requirement already satisfied: Pillow==9.3.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 30)) (9.3.0)
Requirement already satisfied: protobuf==3.19.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 31)) (3.19.6)
Requirement already satisfied: pyasn1==0.4.8 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 32)) (0.4.8)
Requirement already satisfied: pyasn1-modules==0.2.8 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 33)) (0.2.8)
Requirement already satisfied: pyparsing==3.0.9 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 34)) (3.0.9)
Requirement already satisfied: python-dateutil==2.8.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 35)) (2.8.2)
Requirement already satisfied: pytz==2022.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 36)) (2022.6)
Requirement already satisfied: PyYAML==6.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 37)) (6.0)
Requirement already satisfied: requests==2.28.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 38)) (2.28.1)
Requirement already satisfied: requests-oauthlib==1.3.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 39)) (1.3.1)
Requirement already satisfied: rsa==4.9 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 40)) (4.9)
Requirement already satisfied: six==1.16.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 41)) (1.16.0)
Requirement already satisfied: tensorboard==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 42)) (2.11.0)
Requirement already satisfied: tensorboard-data-server==0.6.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 43)) (0.6.1)
Requirement already satisfied: tensorboard-plugin-wit==1.8.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 44)) (1.8.1)
Requirement already satisfied: tensorflow==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 45)) (2.11.0)
Requirement already satisfied: tensorflow-io==0.28.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 46)) (0.28.0)
Requirement already satisfied: tensorflow-estimator==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 47)) (2.11.0)
Requirement already satisfied: termcolor==2.1.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 48)) (2.1.1)
Requirement already satisfied: torch==1.13.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 49)) (1.13.0+cpu)
Requirement already satisfied: torchaudio==0.13.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 50)) (0.13.0+cpu)
Requirement already satisfied: torchvision==0.14.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 51)) (0.14.0+cpu)
Requirement already satisfied: typing_extensions==4.4.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 52)) (4.4.0)
Requirement already satisfied: urllib3==1.26.12 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 53)) (1.26.12)
Requirement already satisfied: Werkzeug==2.2.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 54)) (2.2.2)
Requirement already satisfied: wrapt==1.14.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 55)) (1.14.1)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from astunparse==1.6.3->-r dlio_benchmark/requirements.txt (line 6)) (0.40.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/site-packages (from tensorboard==2.11.0->-r dlio_benchmark/requirements.txt (line 42)) (65.5.0)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from tensorflow==2.11.0->-r dlio_benchmark/requirements.txt (line 45)) (0.28.0)
[kmannthey@pgh-echo-1-1 storage]$

When I try and run datagen I get errors about hydra. 

[kmannthey@pgh-echo-1-1 storage]$ ./benchmark.sh datagen --workload unet3d --num-parallel 8 --param dataset.num_subfolders_train=10 --param dataset.data_folder=unet3d_data
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
[kmannthey@pgh-echo-1-1 storage]$

hydra-core is installed. 

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: hydra-core in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (1.2.0)
Requirement already satisfied: omegaconf~=2.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from hydra-core) (2.2.3)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from hydra-core) (4.9.3)
Requirement already satisfied: packaging in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from hydra-core) (21.3)
Requirement already satisfied: PyYAML>=5.1.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from omegaconf~=2.2->hydra-core) (6.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from packaging->hydra-core) (3.0.9)
[kmannthey@pgh-echo-1-1 storage]$

Any ideas?

kmannthey commented 1 year ago

On my system we need to use "python3.10" not "python3". I did a find and replace in benchmark.sh (python3 to python3.10) and datagen now works.