leggerf
commented
2 years ago Another thing, benchmark preset that need to run on GPU (obviously) fail:
2022-06-14 08:50:05.014250: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib 2022-06-14 08:50:05.014285: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2022-06-14 08:50:06.613821: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib No GPU found. Aborting. 2022-06-14 08:50:06.613849: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303) 2022-06-14 08:50:06.613868: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az127-820): /proc/driver/nvidia/version does not exist "tensorflow_benchmark" preset "cifar_inference" stats match failed: No match for stat "total_time" in benchmark output WARNING: Stats decode for benchmark "tensorflow_benchmark" failed
for example see the tensorflow benchmark output here
We might have discussed this already, but clearly as it is the benchmark is not really useful
We should test in CI when a preset is added/modified