Open laytonjbgmail opened 2 years ago
hariharan-devarajan
commented
2 years ago I am sorry, you didnt miss anything, the setup.py has issues.
I would recommend you clone the repo and check the script https://github.com/hariharan-devarajan/dlio_benchmark/blob/master/scripts/cosmic.sh to see how to use it.
laytonjbgmail
commented
2 years ago Thanks for the quick response. I cloned the repo to my workstation. Then I tried running setup.py again and got the following:
$ python ./setup.py install
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/dist.py:772: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running install
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
creating dlio.egg-info
writing dlio.egg-info/PKG-INFO
writing dependency_links to dlio.egg-info/dependency_links.txt
writing top-level names to dlio.egg-info/top_level.txt
writing manifest file 'dlio.egg-info/SOURCES.txt'
reading manifest file 'dlio.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'dlio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/dlio-0.0.1-py3.9.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dlio-0.0.1-py3.9.egg
Removing /home/laytonjb/anaconda3/lib/python3.9/site-packages/dlio-0.0.1-py3.9.egg
Copying dlio-0.0.1-py3.9.egg to /home/laytonjb/anaconda3/lib/python3.9/site-packages
dlio 0.0.1 is already the active version in easy-install.pth
Installed /home/laytonjb/anaconda3/lib/python3.9/site-packages/dlio-0.0.1-py3.9.egg
Processing dependencies for dlio==0.0.1
Finished processing dependencies for dlio==0.0.1It had the same issues as my previous post.
I then tried going into the source on my workstation and try running dlio_benchmark.py. I got the following output that is the same as my previous post.
$ python ./dlio_benchmark.py -h
Traceback (most recent call last):
File "/home/laytonjb/NVIDIA/ANL/dlio_benchmark/src/./dlio_benchmark.py", line 15, in <module>
from src.common.enumerations import Profiler
ModuleNotFoundError: No module named 'src'Would it be better if I put all Python source files in one directory so that dlio_benchmark.py can find the module it needs? This seems drastic and very messy.
Thanks!
GG
hariharan-devarajan
commented
2 years ago you need to set the PYTHONPATH i believe.
I usually do this.
DLIO_ROOT=<Set your path to DLIO>
export PYTHONPATH=$DLIO_ROOT:$PYTHONPATHThe you can run dlio_benchmark.py
python ${DLIO_ROOT}/src/dlio_benchmark.pyOk i tried a fresh environment and added the instructions into the README for your reference as well
git clone https://github.com/hariharan-devarajan/dlio_benchmark
cd dlio_benchmark/
python3 -m venv ./venv
source venv/bin/activate
pip install -r requirements.txt
export PYTHONPATH=$PWD/src:$PYTHONPATH
python ./src/dlio_benchmark.py -hI also fixed the setup.py now
you can install it by
python setup.py install
# this install dlio_benchmark as an executable.
dlio_benchmark -hWe're u able to test this for you @laytonjbgmail .
MartinHilgeman
commented
1 year ago The dlio_benchmark executable does not work for me:
[martinh@zlogin01 dlio_benchmark]$ python3 setup.py install --prefix=$HOME/.local
running install
running bdist_egg
running egg_info
writing src/dlio.egg-info/PKG-INFO
writing dependency_links to src/dlio.egg-info/dependency_links.txt
writing entry points to src/dlio.egg-info/entry_points.txt
writing top-level names to src/dlio.egg-info/top_level.txt
reading manifest file 'src/dlio.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'src/dlio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/dlio-0.0.1-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dlio-0.0.1-py3.7.egg
Removing /home/martinh/.local/lib/python3.7/site-packages/dlio-0.0.1-py3.7.egg
Copying dlio-0.0.1-py3.7.egg to /home/martinh/.local/lib/python3.7/site-packages
dlio 0.0.1 is already the active version in easy-install.pth
Installing dlio_benchmark script to /home/martinh/.local/bin
Installed /home/martinh/.local/lib/python3.7/site-packages/dlio-0.0.1-py3.7.egg
Processing dependencies for dlio==0.0.1
Finished processing dependencies for dlio==0.0.1
Requirement already satisfied: horovod[tensorflow]>=0.19.5 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.26.1)
Requirement already satisfied: tensorflow>=2.2.0 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 2)) (2.2.0)
Requirement already satisfied: numpy>=1.19.1 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 3)) (1.19.1)
Requirement already satisfied: h5py~=2.10.0 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 4)) (2.10.0)
Requirement already satisfied: pandas>=1.1.3 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 5)) (1.3.5)
Requirement already satisfied: mpi4py>=3.1.3 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 6)) (3.1.3)
Requirement already satisfied: pyyaml in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (6.0)
Requirement already satisfied: cffi>=1.4.0 in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (1.15.1)
Requirement already satisfied: cloudpickle in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (2.2.0)
Requirement already satisfied: packaging in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (21.3)
Requirement already satisfied: psutil in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (5.9.2)
Requirement already satisfied: termcolor>=1.1.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.0.1)
Requirement already satisfied: google-pasta>=0.1.8 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.2.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.3.0)
Requirement already satisfied: six>=1.12.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.16.0)
Requirement already satisfied: absl-py>=0.7.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.3.0)
Requirement already satisfied: grpcio>=1.8.6 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.49.1)
Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.2.0)
Requirement already satisfied: astunparse==1.6.3 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.6.3)
Requirement already satisfied: scipy==1.4.1 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.4.1)
Requirement already satisfied: wrapt>=1.11.1 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.12.1)
Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.2.2)
Requirement already satisfied: wheel>=0.26 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.37.1)
Requirement already satisfied: gast==0.3.3 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.3.3)
Requirement already satisfied: keras-preprocessing>=1.1.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.1.2)
Requirement already satisfied: protobuf>=3.8.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.20.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /cm/local/apps/python37/lib/python3.7/site-packages (from pandas>=1.1.3->-r ./requirements.txt (line 5)) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /cm/local/apps/python37/lib/python3.7/site-packages (from pandas>=1.1.3->-r ./requirements.txt (line 5)) (2021.1)
Requirement already satisfied: pycparser in /home/martinh/.local/lib/python3.7/site-packages (from cffi>=1.4.0->horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (2.21)
Requirement already satisfied: werkzeug>=0.11.15 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.2.2)
Requirement already satisfied: setuptools>=41.0.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (57.4.0)
Requirement already satisfied: requests<3,>=2.21.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.26.0)
Requirement already satisfied: markdown>=2.6.8 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.4.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.35.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.4.6)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.8.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/martinh/.local/lib/python3.7/site-packages (from packaging->horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (3.0.9)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/martinh/.local/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (4.2.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /home/martinh/.local/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/martinh/.local/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: importlib-metadata>=4.4 in /cm/local/apps/python37/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (4.8.1)
Requirement already satisfied: certifi>=2017.4.17 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2021.5.30)
Requirement already satisfied: idna<4,>=2.5 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.2)
Requirement already satisfied: charset-normalizer~=2.0.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.25.11)
Requirement already satisfied: MarkupSafe>=2.1.1 in /home/martinh/.local/lib/python3.7/site-packages (from werkzeug>=0.11.15->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.1.1)
Requirement already satisfied: typing-extensions>=3.6.4 in /cm/local/apps/python37/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.10.0.2)
Requirement already satisfied: zipp>=0.5 in /cm/local/apps/python37/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.5.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/martinh/.local/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.4.8)
Requirement already satisfied: oauthlib<3.0.0,>=2.1.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.1.0)
[martinh@zlogin01 dlio_benchmark]$ dlio_benchmark
Traceback (most recent call last):
File "/home/martinh/.local/bin/dlio_benchmark", line 33, in <module>
sys.exit(load_entry_point('dlio==0.0.1', 'console_scripts', 'dlio_benchmark')())
File "/home/martinh/.local/bin/dlio_benchmark", line 25, in importlib_load_entry_point
return next(matches).load()
File "/cm/local/apps/python37/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 194, in load
module = import_module(match.group('module'))
File "/cm/local/apps/python37/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dlio_benchmark'The only way to get it to work is:
[martinh@zlogin01 dlio_benchmark]$ pwd
/home/martinh/work/dlio_benchmark
[martinh@zlogin01 dlio_benchmark]$ export PYTHONPATH=/home/martinh/work/dlio_benchmark
[martinh@zlogin01 dlio_benchmark]$ python ./src/dlio_benchmark.py -h
usage: dlio_benchmark.py [-h] [-f {tfrecord,hdf5,csv,npz,hdf5_opt}]
[-r {off,seed,random}] [-ms SHUFFLE_SIZE]
[-m {off,seed,random}] [-rt {memory,on_demand}]
[-fa {multi,shared,collective}] [-rl RECORD_LENGTH]
[-nf NUM_FILES] [-sf NUM_SAMPLES] [-bs BATCH_SIZE]
[-e EPOCHS] [-se SEED_CHANGE_EPOCH]
[-gd GENERATE_DATA] [-df DATA_FOLDER]
[-of OUTPUT_FOLDER] [-fp FILE_PREFIX]
[-go GENERATE_ONLY] [-k KEEP_FILES] [-p PROFILING]
[-l LOGDIR] [-s SEED] [-c CHECKPOINT]
[-sc STEPS_CHECKPOINT] [-ts TRANSFER_SIZE]
[-tr READ_THREADS] [-tc COMPUTATION_THREADS]
[-ct COMPUTATION_TIME] [-rp PREFETCH]
[-ps PREFETCH_SIZE] [-ec ENABLE_CHUNKING]
[-cs CHUNK_SIZE] [-co {none,gzip,lzf,bz2,zip,xz}]
[-cl COMPRESSION_LEVEL] [-d DEBUG]
DLIO Benchmark
...We have migrated the repo to alcf located at https://github.com/argonne-lcf/dlio_benchmark. Can you please check that repo and if issue persists please raise the issue there. Sorry for the inconvenience.
Hello,
I install dlio using the following method
The installation output is ,
I wanted to then run the command
python ./dlio_benchmark.pybut I cannot find the script. anywhere.I tried going to the source to run
dlio_benchmark.py, python3 ./dlio_benchmark.pyand I get the following output,I apologize for missing something obvious - could someone help me?
BTW - Python3 3.9.12, horovod 0.25.0, Tensorflow 2.9.1, HDF5 3.7.0, numpy 1.21.6
Thank you.