ModuleNotFoundError: No module named 'official.benchmark'

[gaborvecsei]

I wanted to run some benchmarks on my local machine and the problem happened to me which you can see at the title.

• I created a docker image with: python3 benchmarks/perfzero/lib/setup.py --tensorflow_pip_spec=tensorflow==2.3.0
• Then I ran the docker image: docker run -it --rm --name tf-test -v $(pwd):/workspace --gpus all perfzero/tensorflow bash

2020-08-29 08:45:27,217 INFO: The following benchmark methods will be executed: ['official.benchmark.keras_cifar_benchmark.Resnet56KerasBenchmarkSynth.benchmark_1_gpu_no_dist_strat']
Setup complete. Running 1 trials
Running trial 1 / 1
2020-08-29 08:45:27,227 INFO: Created directory /workspace/perfzero/workspace/output/2020-08-29-08-45-27-227353
2020-08-29 08:45:27,227 INFO: Created directory /workspace/perfzero/workspace/output/2020-08-29-08-45-27-227353
2020-08-29 08:45:27,231 ERROR: Benchmark execution for official.benchmark.keras_cifar_benchmark.Resnet56KerasBenchmarkSynth.benchmark_1_gpu_no_dist_strat failed due to error:
 Traceback (most recent call last):
  File "/workspace/perfzero/lib/perfzero/benchmark_method_runner.py", line 101, in _run_internal
    constructor_args=constructor_args)
  File "/workspace/perfzero/lib/perfzero/utils.py", line 451, in instantiate_benchmark_class
    module = importlib.import_module(module_import_path)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'official.benchmark'

I can see that https://github.com/tensorflow/models does not have official.benchmark module, but then how should I run the benchmark? Could you update the README.md

[lindong28]

@gaborvecsei @reedwm I am sorry for late reply. Just noticed this issue...

I think the issue is because https://github.com/tensorflow/models has been refactored such that the official.benchmark module only exists in the benchmark branch but not the master branch.

This issue can be fixed by running PerfZero with --git_repos="https://github.com/tensorflow/models.git;benchmark" so that PerfZero will checkout the right branch of this repository.

@gaborvecsei, could you re-try after making the change described above? I will update the README afterwards.

[gaborvecsei]

@lindong28 Thank you for your response! Now I can run the benchmark. I only tried with the following setup:

python3 /workspace/perfzero/lib/benchmark.py \
    --git_repos="https://github.com/tensorflow/models.git;benchmark"
    --python_path=models \
    --gcloud_key_file_url="" \
    --benchmark_methods=official.benchmark.keras_cifar_benchmark.Resnet56KerasBenchmarkSynth.benchmark_1_gpu_no_dist_strat

[lindong28]

@gaborvecsei Thanks for the confirmation! The PerfZero README has been updated to use the right repository link.