Open gaborvecsei opened 3 years ago
@reedwm Toby is the person who added Docker support for PerfZero. I don't have experience with docker and I don't know which part of service in our infra uses docker. I don't know who is able to maintain this feature now that Toby has left this project.
@lindong28 I can look into the code and create a PR with the necessary changes
Thank you @gaborvecsei for offering to fix this issue!
If the PR is easy to review (e.g. it just changed a version), it will be great and I can just approve it. If the PR involves something that requires docker expertise, I will ask around and see who can help with this.
I have the same problem. I tried different containers nothing worked. Also tried to install the version named in the dockerfile. Nothing really helped that the benchmark is run on the gpu. Only the cpu is used.
python3 perfzero/lib/setup.py --dockerfile_path=docker/Dockerfile_ubuntu_1804_tf_v2
python3 perfzero/lib/setup.py --dockerfile_path=docker/Dockerfile_ubuntu_1804_tf_v2 --tensorflow_pip_spec=tensorflow-gpu==2.1.0
nvidia-docker run -it --rm -v $(pwd):/workspace -v /data:/data perfzero/tensorflow bash
Then i run the benchmark the error is similar:
root@cb1e8eb587b0:/# python3 /workspace/perfzero/lib/benchmark.py --git_repos="https://github.com/tensorflow/models.git;benchmark" --python_path=models --gcloud_key_file_url="" --benchmark_methods=official.benchmark.keras_cifar_benchmark.Resnet56KerasBenchmarkSynth.benchmark_1_gpu_no_dist_strat
Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2021-04-19 06:35:56.185318: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-04-19 06:35:56.185416: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-04-19 06:35:56.185429: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Then i seraching for the libcuadart* i get:
root@cb1e8eb587b0:/# find / -name "libcudart.so.*"
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so.10.1
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so.10.1.243
@gaborvecsei You mentioned it should be only the false version. Did you find a version that is still working and can you share the solution you find.
Any other way to solve the problem?
Having the same issue
@GabrielGardin
I found out that for me the dockerfile with Ubuntu 18.04 and Cuda 11.0 works, when using tensorflow version 2.4.
The command to build the docker looks like this:
python3 perfzero/lib/setup.py --dockerfile_path=docker/Dockerfile_ubuntu_1804_tf_cuda_11_0 --tensorflow_pip_spec=tensorflow==2.4
Steps to reproduce the error:
python perfzero/lib/setup.py --tensorflow_pip_spec=tensorflow==2.3.0
docker run -it --gpus all --rm -v $(pwd):/workspace perfzero/tensorflow bash
python3 /workspace/perfzero/lib/benchmark.py --git_repos="https://github.com/tensorflow/models.git;benchmark" --python_path=models --gcloud_key_file_url="" --benchmark_methods=official.benchmark.keras _cifar_benchmark.Resnet56KerasBenchmarkSynth.benchmark_1_gpu_no_dist_strat
The benchmark starts but only on CPUs because of the error:
When I am searching for the "
libcudart.so.*
" (find / -name "libcudart.so.*"
) the results are the following:So only the wrong version is installed.