Convert ResNet50 model

[psyhtest]

See the instructions here.

[psyhtest]

At the very least, we will need to distinguish between different source models, because the conversion script currently has its parameters hardcoded for SSD-MobileNet. For example, SSD-MobileNet has CK_ENV_TENSORFLOW_MODEL_MODEL_NAME='MLPerf SSD-MobileNet' set in the environment. Similarly, ResNet50 v1.5 has CK_ENV_TENSORFLOW_MODEL_NAME="MLPerf ResNet50 v1.5" set.

[psyhtest]

By the way, here's a subset of COCO images that should be used for calibration.

[psyhtest]

Curiously, Intel don't mention any subsets:

Model calibration requires a validation dataset (to keep track of the accuracy during calibration). Currently, the calibration tool comes with example support of classification and object detection models on ImageNet and VOC2007/COCO data sets respectively and associated accuracy metrics. It is relatively straightforward to add another datasets and metrics.

[psyhtest]

This appears to be a slightly different version of calibration instructions with more ResNet50-specific instructions.

[G4V]

In the instructions as linked to in the first post, the link within to the calibration tool is broken.

Good link here -

https://docs.openvinotoolkit.org/2019_R1/_inference_engine_tools_calibration_tool_README.html

[psyhtest]

Thanks @G4V!

One gotcha is that the variation is called resnet50, not resnet:

anton@velociti:~$ ck install package --tags=model,resnet,openvino
CK error: [package] package with tags "model,resnet,openvino" for your environment was not found!!

$ ck install package --tags=model,openvino,resnet50
...
/usr/bin/python3.6 /home/anton/CK_TOOLS/lib-openvino-gcc-8.3.0-2019_R3.1-linux-64/dldt/model-optimizer/mo.py --model_name converted_model --input_model /home/anton/CK_TOOLS/model-tf-mlperf-resnet-downloaded/resnet50_v1.pb --input_shape [1,224,224,3]
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      /home/anton/CK_TOOLS/model-tf-mlperf-resnet-downloaded/resnet50_v1.pb
        - Path for generated IR:        /home/anton/CK_TOOLS/model-openvino-converted-from-tf-resnet50/.
        - IR output name:       converted_model
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         [1,224,224,3]
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  None
Model Optimizer version:        unknown version
[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: /home/anton/CK_TOOLS/model-openvino-converted-from-tf-resnet50/./converted_model.xml
[ SUCCESS ] BIN file: /home/anton/CK_TOOLS/model-openvino-converted-from-tf-resnet50/./converted_model.bin
[ SUCCESS ] Total execution time: 25.70 seconds.
Done.

That's fine. We can always rename it (or duplicate).

Another gotcha is that if you don't have the original model installed, you are given a choice of three models:

$ ck clean env --tags=model,tf,mlperf,resnet
$ ck install package --tags=model,resnet50,openvino
...
 0) model-tf-ssd-resnet50-fpn-coco  Version 20180703  (09baac5e6f931db2)
 1) model-tf-mlperf-resnet  Version 1.5  (39aa18439d90c227)
 2) model-tf-faster-rcnn-resnet50-lowproposals-coco  Version reference  (640458144a59763d)

We should add some unique tags to disambiguate it. For example, the desired choice 1) has the following tags:

  "tags": [
    "model",
    "image-classification",
    "tf",
    "nhwc",
    "resnet",
    "resnet50",
    "resnet50-v1.5",
    "v1.5",
    "mlperf",
    "weights",
    "frozen",
    "downloaded"
  ],

I think two unique tags here are mlperf and image-classification.

[G4V]

Although the documentation for 2019r3 still refers to it, it looks like the command "convert_annotation" has been removed.

I'm guessing that this is the same thing - https://github.com/opencv/open_model_zoo/pull/186/files

For the current version of OpenVINO 2020.1/2 it mentions calibration, but there's no calibration tool listed under the tools section. https://docs.openvinotoolkit.org/2020.1/_docs_IE_DG_Tools_Overview.html

Some digging will be required to find out exactly what's needed to run calibration.

Possibly the open_model_zoo repo will be needed. This looks to be available with the 2020 OpenVINO distro.

[psyhtest]

Let's stick to 2019_R3.1 for now. I guess this is the tool?

anton@velociti:~/CK_TOOLS$ find ~ -name convert_annotation.py
/home/anton/CK_TOOLS/lib-openvino-gcc-8.3.0-2019_R3.1-linux-64/dldt/inference-engine/tools/accuracy_checker_tool/convert_annotation.py

[psyhtest]

$ tree /home/anton/CK_TOOLS/lib-openvino-gcc-8.3.0-2019_R3.1-linux-64/dldt/inference-engine/tools/accuracy_checker_tool/
/home/anton/CK_TOOLS/lib-openvino-gcc-8.3.0-2019_R3.1-linux-64/dldt/inference-engine/tools/accuracy_checker_tool/
├── accuracy_check.py
├── convert_annotation.py
└── README.md

0 directories, 3 files

[psyhtest]

If you install a small subset of ImageNet (500 images):

$ ck install package --tags=dataset,imagenet,2012,val,original,min --no_tags=resized

you should be able to run the program as follows:

$ ck compile program:mlperf-inference-v0.5
$ export NPROCS=`grep -c processor /proc/cpuinfo`
$ ck run program:mlperf-inference-v0.5 --skip_print_timers \
--cmd_key=image-classification --env.CK_LOADGEN_TASK=image-classification \
--env.CK_LOADGEN_SCENARIO=Offline --env.CK_LOADGEN_MODE=Accuracy  \
--env.CK_OPENVINO_NTHREADS=$NPROCS --env.CK_OPENVINO_NSTREAMS=$NPROCS --env.CK_OPENVINO_NIREQ=$NPROCS

This will probably fail though with:

ValueError: cannot convert float NaN to integer
Traceback (most recent call last):
  File "/home/anton/CK/ck/kernel.py", line 10820, in <module>
    r=access(sys.argv[1:])
  File "/home/anton/CK/ck/kernel.py", line 10776, in access
    rr=perform_action(i)
  File "/home/anton/CK/ck/kernel.py", line 4126, in perform_action
    return a(i)
  File "/home/anton/CK_REPOS/ck-autotuning/module/program/module.py", line 3571, in run
    run_output_dict = process(i)
  File "/home/anton/CK_REPOS/ck-autotuning/module/program/module.py", line 182, in process
    r=process_in_dir(ii)
  File "/home/anton/CK_REPOS/ck-autotuning/module/program/module.py", line 3042, in process_in_dir
    rxx=cs.ck_postprocess(ii)
  File "/home/anton/CK_REPOS/ck-mlperf/script/image-classification/loadgen_postprocess.py", line 76, in ck_postprocess
    output = check_output(command).decode('ascii')
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '[u'/usr/bin/python3.6', u'/home/anton/CK_TOOLS/mlperf-inference-dividiti.v0.5-intel/inference/v0.5/classification_and_detection/tools/accuracy-imagenet.py', '--mlperf-accuracy-file', 'mlperf_log_accuracy.json', '--imagenet-val-file', u'/home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-aux/val.txt', '--dtype', 'float32']' returned non-zero exit status 1

If we look at the accuracy log:

$ head `ck find program:mlperf-inference-v0.5`/tmp/mlperf_log_accuracy.json
[
{ "seq_id" : 2, "qsl_idx" : 364, "data" : "FFFFFFFF" },
{ "seq_id" : 13, "qsl_idx" : 243, "data" : "FFFFFFFF" },
{ "seq_id" : 14, "qsl_idx" : 216, "data" : "FFFFFFFF" },
{ "seq_id" : 5, "qsl_idx" : 369, "data" : "FFFFFFFF" },
{ "seq_id" : 1, "qsl_idx" : 348, "data" : "FFFFFFFF" },
{ "seq_id" : 10, "qsl_idx" : 303, "data" : "FFFFFFFF" },
{ "seq_id" : 11, "qsl_idx" : 471, "data" : "FFFFFFFF" },
{ "seq_id" : 0, "qsl_idx" : 254, "data" : "FFFFFFFF" },
{ "seq_id" : 6, "qsl_idx" : 209, "data" : "FFFFFFFF" },

it is clear that the returned values (which are supposed to be class predictions) are all FFFFFFFF (hence the cannot convert float NaN to integer error).

It's probably because the conversion result is an FP32 IR. That's why the calibration is needed.

[G4V]

Cheers. User error searching for the convert_annotation.py in the dldt directory. I'd assumed it had been removed and replaced by something else.

[G4V]

Current progress on calibration setup -

To manually run convert_annotation

Python script found here - ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/accuracy_checker_tool

Running convert_annotation.py requires accuracy_checker which is located in open_model_zoo, 2019 r3.1 found here -

https://github.com/opencv/open_model_zoo/releases/tag/2019_R3.1

Download the tarball and extract. To install run - ./tools/accuracy_checker/setup.py install

Then run convert_annotation.py with the following command line -

python3 convert_annotation.py imagenet --annotation_file ~/CK-TOOLS/dataset-imagenet-ilsvrc2012-val-min/val_map.txt --labels_file ~/CK-TOOLS/dataset-imagenet-ilsvrc2012-aux/synset_words.txt --has_background True This produces two files imagenet.pickle and imagenet.json.

Next step is to run calibrate.py. The script is found here -

~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/calibration_tool

Command line -

python3.6 calibrate.py -c ../accuracy_checker_tool/resnet_v1.5_50.yml -M ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/model-optimizer/ -C . ~/CK-TOOLS/model-openvino-converted-from-tf-lib.openvino-2019_R3.1-resnet50 --output_dir /stuff

The script relies on the following modules 1) openvino.tools.calibration, found at -

~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/tools/calibration

2) openvino.inference_engine

This is a python wrapper around the inference engine and requires openvino to be built with python enabled (fails to build for 3.7) -

  cmake -DENABLE_PYTHON=ON -DPYTHON_EXECUTABLE=`which python3.6` \
    -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.6m.so \
    -DPYTHON_INCLUDE_DIR=/usr/include/python3.6 ..

See ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/ie_bridges/python/README.md for more info.

This places the inference_engine shim in -

~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/bin/intel64/Release/lib/python_api/python3.6/openvino/inference_engine To bring together into something that can be executed -

cd ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/calibration_tool
mkdir openvino
cd openvino
ln -s ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/bin/intel64/Release/lib/python_api/python3.6/openvino/inference_engine
ln -s ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/tools/
cd tools ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/bin/intel64/Release/lib/python_api/python3.6/openvino/tools/statistics_collector/

I had to install these -

python3.6 -m pip install xmltodict
python3.6 -m pip install progress
python3.6 -m pip install networkx==2.3
python3.6 -m pip install test-generator

After all this, running the above calibrate.py command line almost competes successfully.

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: /home/gavin/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/calibration_tool/./resnet50_v1.xml
[ SUCCESS ] BIN file: /home/gavin/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/calibration_tool/./resnet50_v1.bin
[ SUCCESS ] Total execution time: 15.03 seconds. 
Traceback (most recent call last):
  File "calibrate.py", line 20, in <module>
    with calibration.CommandLineProcessor.process() as config:
  File "/home/gavin/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/calibration_tool/openvino/tools/calibration/command_line_processor.py", line 140, in process
    cpu_extension = DLSDKLauncher.get_cpu_extension(launcher['cpu_extensions'], args.cpu_extensions_mode)
  File "/usr/local/lib/python3.6/dist-packages/accuracy_checker-0.7.3-py3.6.egg/accuracy_checker/launcher/dlsdk_launcher.py", line 418, in get_cpu_extension
    raise ConfigError('suitable CPU extension lib not found in {}'.format(extensions_path))
accuracy_checker.config.config_validator.ConfigError: suitable CPU extension lib not found in /home/gavin/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/calibration_tool

Note that this is all very hand coded but at least validates that there's a route to calibrating the models.

[G4V]

Contents of the yml file - again hard coded -

models:
  - name: ResNet_v1.5_50
    launchers:
      - framework: dlsdk
        device: CPU
        tf_model: ../../../../../../../../../home/gavin/CK-TOOLS/model-tf-mlperf-resnet-downloaded/resnet50_v1.pb
        adapter: classification
        mo_params:
          data_type: FP32
          input_shape: (1, 224, 224, 3)
          output: softmax_tensor
        cpu_extensions: AUTO
    datasets:
      - name: ImageNet2012_bkgr
        data_source: ~/CK-TOOLS/dataset-imagenet-ilsvrc2012-val-min/
        annotation: ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/accuracy_checker_tool/imagenet.pickle
        dataset_meta: ~/CK-TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/inference-engine/tools/accuracy_checker_tool/imagenet.json
        annotation_conversion:
          converter: imagenet
          annotation_file: ~/CK-TOOLS/dataset-imagenet-ilsvrc2012-val-min/val_map.txt
          labels_file: ~/CK-TOOLS/dataset-imagenet-ilsvrc2012-aux/synset_words.txt
          has_background: True
        subsample_size: 500
        preprocessing:
          - type: resize
            size: 256
            aspect_ratio_scale: greater
          - type: crop
            size: 224
          - type: normalization
            mean: 123, 117, 104
        metrics:
          - name: accuracy @ top1
            type: accuracy
            top_k: 1
          - name: accuracy @ top5
            type: accuracy
            top_k: 5

[psyhtest]

@G4V I run inference on the full ImageNet validation set using the ResNet50 files you shared. Unfortunately, the resulting accuracy was only 0.1%:

$ cat ~/CK_REPOS/ck-openvino/program/mlperf-inference-v0.5/tmp/accuracy.txt
accuracy=0.100%, good=50, total=50000

However, I seem to get exactly the same if I use any of the logs from Intel's submission e.g.:

/usr/bin/python3.6 \
/home/anton/CK_TOOLS/mlperf-inference-dividiti.v0.5-intel/inference/v0.5/classification_and_detection/tools/accuracy-imagenet.py \
--mlperf-accuracy-file /home/anton/projects/mlperf/inference_results_v0.5_dividiti/closed/Intel/results/ICL-I3-1005G1_OpenVINO-Windows/resnet/Offline/accuracy/mlperf_log_accuracy.json \
--imagenet-val-file /home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-aux/val.txt --dtype float32
accuracy=0.100%, good=50, total=50000

[psyhtest]

anton@diviniti:~/projects/mlperf/inference_results_v0.5_dividiti/closed/Intel/results$ for mlperf_log_accuracy_json in \
  ./ICL-I3-1005G1_OpenVINO-Windows/resnet/Offline/accuracy/mlperf_log_accuracy.json \
  ./ICL-I3-1005G1_OpenVINO-Windows/resnet/SingleStream/accuracy/mlperf_log_accuracy.json \
  ./nnpi-1000-2x_onnx/resnet/Offline/accuracy/mlperf_log_accuracy.json \
  ./nnpi-1000-2x_onnx/resnet/Server/accuracy/mlperf_log_accuracy.json \
  ./clx-9282-2s_pytorch-caffe2/resnet/Offline/accuracy/mlperf_log_accuracy.json \
  ./clx-9282-2s_pytorch-caffe2/resnet/SingleStream/accuracy/mlperf_log_accuracy.json \
 ./clx-9282-2s_pytorch-caffe2/resnet/Server/accuracy/mlperf_log_accuracy.json \
; do \
  /usr/bin/python3.6 \
  /home/anton/CK_TOOLS/mlperf-inference-dividiti.v0.5-intel/inference/v0.5/classification_and_detection/tools/accuracy-imagenet.py \
  --mlperf-accuracy-file $mlperf_log_accuracy_json \
  --imagenet-val-file /home/anton/CK_TOOLS/dataset-imagenet-ilsvrc2012-aux/val.txt \
  --dtype float32 \
; done
accuracy=0.100%, good=50, total=50000
accuracy=0.100%, good=50, total=50000
accuracy=0.100%, good=50, total=50000
accuracy=0.100%, good=50, total=50000
accuracy=0.100%, good=50, total=50000
accuracy=0.100%, good=50, total=50000
accuracy=0.100%, good=50, total=50000

[psyhtest]

anton@diviniti:~/projects/mlperf/inference_results_v0.5_dividiti/closed/Intel/results$ for accuracy_txt in \
  ./ICL-I3-1005G1_OpenVINO-Windows/resnet/Offline/accuracy/accuracy.txt \
  ./ICL-I3-1005G1_OpenVINO-Windows/resnet/SingleStream/accuracy/accuracy.txt \
  ./nnpi-1000-2x_onnx/resnet/Offline/accuracy/accuracy.txt \
  ./nnpi-1000-2x_onnx/resnet/Server/accuracy/accuracy.txt \
  ./clx-9282-2s_pytorch-caffe2/resnet/Offline/accuracy/accuracy.txt \
  ./clx-9282-2s_pytorch-caffe2/resnet/SingleStream/accuracy/accuracy.txt \
 ./clx-9282-2s_pytorch-caffe2/resnet/Server/accuracy/accuracy.txt \
; do \
  cat $accuracy_txt \
; done
accuracy=76.400%, good=38200, total=50000
accuracy=76.234%, good=38117, total=50000
accuracy=75.720%, good=37860, total=50000
accuracy=75.720%, good=37860, total=50000
accuracy=75.766%, good=37883, total=50000
accuracy=76.124%, good=38062, total=50000
accuracy=76.124%, good=38062, total=50000

[psyhtest]

By the way, the lower bound was (0.99*76.456)

accuracy=75.692%, good=37846, total=50000

The NNPI-1000 result was only 14 predictions above that...

[psyhtest]

A sanity check that I use the latest version of the accuracy-imagenet.py script:

$ wget https://raw.githubusercontent.com/mlperf/inference/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py
--2020-04-20 17:40:06--  https://raw.githubusercontent.com/mlperf/inference/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.60.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.60.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2197 (2.1K) [text/plain]
Saving to: ‘accuracy-imagenet.py’

accuracy-imagenet.py                               100%[================================================================================================================>]   2.15K  --.-KB/s    in 0s      

2020-04-20 17:40:07 (18.3 MB/s) - ‘accuracy-imagenet.py’ saved [2197/2197]

$ md5sum accuracy-imagenet.py 
14ebc71db7376291d2ee62094be3dd3c  accuracy-imagenet.py

$ md5sum /home/anton/CK_TOOLS/mlperf-inference-dividiti.v0.5-intel/inference/v0.5/classification_and_detection/tools/accuracy-imagenet.py
14ebc71db7376291d2ee62094be3dd3c  /home/anton/CK_TOOLS/mlperf-inference-dividiti.v0.5-intel/inference/v0.5/classification_and_detection/tools/accuracy-imagenet.py

[psyhtest]

OK, opened https://github.com/mlperf/inference_results_v0.5/issues/29.

[psyhtest]

Apparently, they return int32 from their model, so that --dtype int32 should be used, not --dtype float32. This image classification postprocessing script now reads this type from the CK_MLPERF_ACCURACY_IMAGENET_TYPE variable, which gets defined as int32 for ResNet50.

[psyhtest]

Thanks to @G4V we now have a good conversion workflow for ResNet50!

Unfortunately, it still fails in some environments. Here's how to reproduce one of the issues via Docker:

$ ck pull repo:ck-mlperf
$ ck build docker:image-classification-openvino.ubuntu-18.04

Due to failing to build some parts of OpenVINO in this environment (GCC 7.5.0, OpenCV 3.4.3) calibration fails with:

######################################################################################
Running calibration ...

LD_LIBRARY_PATH=/home/dvdt/CK_TOOLS/lib-opencv-3.4.3-gcc-7.5.0-linux-64/install/lib::/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/lib/ ; /usr/bin/python3 /home/dvdt/CK_TOOLS/lib-openvin
o-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/calibrate.py -c /home/dvdt/CK_TOOLS/model-openvino-converted-from-tf-lib.openvino-pre-release-resnet50/config.yml -M /home/dvd
t/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/model-optimizer -C /home/dvdt/CK_TOOLS/model-openvino-converted-from-tf-lib.openvino-pre-release-resnet50 -e /home/dvdt/CK_TOOLS/lib-openvino-gc
c-7.5.0-pre-release-linux-64/lib/ --output_dir /home/dvdt/CK_TOOLS/model-openvino-converted-from-tf-lib.openvino-pre-release-resnet50
Traceback (most recent call last):
  File "/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/calibrate.py", line 17, in <module>
    import openvino.tools.calibration as calibration
  File "/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/openvino/tools/calibration/__init__.py", line 16, in <module>
    from .aggregated_statistics import AggregatedStatistics
  File "/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/openvino/tools/calibration/aggregated_statistics.py", line 19, in <module>
    import openvino.inference_engine as ie
ModuleNotFoundError: No module named 'openvino.inference_engine'
Error: Calibration failed!

Enter the failed image:

$ docker image ls | grep none | head -n 1
<none>                                                                                <none>                                            e806f98dc1a9        59 minutes ago      5.87GB

$ docker run -it --rm e806f98dc1a9 bash
dvdt@60773dd9235c:~$ cd ~/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/obj/ && make

[100%] Building CXX object ie_bridges/python/src/openvino/tools/statistics_collector/CMakeFiles/statistics_collector_api.dir/statistics_collector_api.cxx.o
/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/obj/ie_bridges/python/src/openvino/tools/statistics_collector/statistics_collector_api.cxx:619:10: fatal error: statistics_processor.hpp: No such file or directory
 #include <statistics_processor.hpp>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
ie_bridges/python/src/openvino/tools/statistics_collector/CMakeFiles/statistics_collector_api.dir/build.make:67: recipe for target 'ie_bridges/python/src/openvino/tools/statistics_collector/CMakeFiles/statistics_collector_api.dir/statistics_collector_api.cxx.o' failed
make[2]: *** [ie_bridges/python/src/openvino/tools/statistics_collector/CMakeFiles/statistics_collector_api.dir/statistics_collector_api.cxx.o] Error 1
CMakeFiles/Makefile2:2777: recipe for target 'ie_bridges/python/src/openvino/tools/statistics_collector/CMakeFiles/statistics_collector_api.dir/all' failed
make[1]: *** [ie_bridges/python/src/openvino/tools/statistics_collector/CMakeFiles/statistics_collector_api.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2

It seems to be the same issue as https://github.com/openvinotoolkit/openvino/issues/324. However, for some environments adding OpenCV helps, but for others it does not.

I have the same failure with GCC 7.5.0 where everything builds just fine with GCC 8.5.0. Maybe we need a newer version of OpenCV.

[psyhtest]

Building the statistics_collector can also fail due to linking, not to compiling:

[ 84%] Linking CXX executable /home/anton/CK_TOOLS/lib-openvino-gcc-8.4.0-pre-release-linux-64/dldt/inference-engine/bin/intel64/Release/statistics_collector
/home/anton/CK_TOOLS/lib-openvino-gcc-8.4.0-pre-release-linux-64/dldt/inference-engine/bin/intel64/Release/lib/libstatistics_collector_s.a(image_decoder.cpp.o): In function `cv::Mat::~Mat()':
image_decoder.cpp:(.text._ZN2cv3MatD2Ev[_ZN2cv3MatD5Ev]+0x53): undefined reference to `cv::fastFree(void*)'
image_decoder.cpp:(.text._ZN2cv3MatD2Ev[_ZN2cv3MatD5Ev]+0x61): undefined reference to `cv::Mat::deallocate()'
/home/anton/CK_TOOLS/lib-openvino-gcc-8.4.0-pre-release-linux-64/dldt/inference-engine/bin/intel64/Release/lib/libstatistics_collector_s.a(image_decoder.cpp.o): In function `cv::Size_<int> addToBlob<unsig
ned char>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, InferenceEngine::Blob&, PreprocessingOptions const&)':
image_decoder.cpp:(.text._Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions[_Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112b
asic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions]+0x397): undefined reference to `cv::imread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const&, int)'
image_decoder.cpp:(.text._Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions[_Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112b
asic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions]+0x514): undefined reference to `cv::fastFree(void*)'
image_decoder.cpp:(.text._Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions[_Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112b
asic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions]+0x10d0): undefined reference to `cv::fastFree(void*)'
image_decoder.cpp:(.text._Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions[_Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112b
asic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions]+0x114f): undefined reference to `cv::fastFree(void*)'
image_decoder.cpp:(.text._Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions[_Z9addToBlobIhEN2cv5Size_IiEERKNSt7__cxx1112b
asic_stringIcSt11char_traitsIcESaIcEEEiRN15InferenceEngine4BlobERK20PreprocessingOptions]+0x124c): undefined reference to `cv::Mat::copySize(cv::Mat const&)'
...

[psyhtest]

In the failing Docker container, there's apparently OpenCV 3.2:

$ docker run -it --rm df348cb3bb73 "apt show libopencv-dev"
Package: libopencv-dev
Version: 3.2.0+dfsg-4ubuntu0.1
Priority: optional
Section: universe/libdevel
Source: opencv
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 734 kB
Depends: libopencv-calib3d-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-contrib-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-core-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-features2d-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-flann-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-highgui-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-imgcodecs-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-imgproc-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-ml-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-objdetect-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-photo-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-shape-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-stitching-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-superres-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-ts-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-video-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-videoio-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-videostab-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv-viz-dev (= 3.2.0+dfsg-4ubuntu0.1), libopencv3.2-java (= 3.2.0+dfsg-4ubuntu0.1), pkg-config, libc6 (>= 2.27), libgcc1 (>= 1:3.0), libopencv-calib3d3.2, libopencv-contrib3.2, libopencv-core3.2, libopencv-features2d3.2, libopencv-highgui3.2, libopencv-imgcodecs3.2, libopencv-imgproc3.2, libopencv-videoio3.2, libstdc++6 (>= 5.2), libtbb2
Recommends: opencv-data
Suggests: opencv-doc
Conflicts: libcv-dev, libcvaux-dev, libhighgui-dev
Breaks: libopencv-core-dev (<= 2.3.1-8)
Replaces: libcv-dev, libcvaux-dev, libhighgui-dev
Homepage: https://opencv.org
Supported: 3y
Download-Size: 223 kB
APT-Sources: http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
Description: development files for opencv

N: There is 1 additional record. Please use the '-a' switch to see it

[psyhtest]

On the failing machine, there's OpenCV 2.4.9:

$ sudo apt show libopencv-dev                                                                                          
[sudo] password for anton: 
Package: libopencv-dev
Version: 2.4.9.1+dfsg-1.5ubuntu1.1
Priority: optional
Section: universe/libdevel
Source: opencv
Origin: Ubuntu
Maintainer: Kubuntu Developers <kubuntu-devel@lists.ubuntu.com>
Original-Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 674 kB
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libopencv-calib3d2.4v5, libopencv-core2.4v5, libopencv-highgui2.4v5, libopencv-imgproc2.4v5, libopencv-ml2.4v5, libopencv-objdetect2.4v5, libstdc++6 (>= 5.2), libopencv-core-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-ml-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-imgproc-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-video-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-objdetect-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-highgui-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-calib3d-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-flann-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-features2d-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-legacy-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-contrib-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-ts-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-photo-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-videostab-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-stitching-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-gpu-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-superres-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv-ocl-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv2.4-java (= 2.4.9.1+dfsg-1.5ubuntu1.1), libopencv2.4-jni (= 2.4.9.1+dfsg-1.5ubuntu1.1), libcv-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libhighgui-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), libcvaux-dev (= 2.4.9.1+dfsg-1.5ubuntu1.1), pkg-config
Recommends: opencv-data
Breaks: libopencv-core-dev (<= 2.3.1-8)
Homepage: http://opencv.org/
Supported: 5y
Download-Size: 169 kB
APT-Sources: http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages

[psyhtest]

And on the machine where everything now works, there's OpenCV 4.1.1:

$ sudo apt show libopencv-dev
[sudo] password for anton:
Package: libopencv-dev
Version: 4.1.1-2-gd5a58aa75
Status: install ok installed
Priority: optional
Section: libs
Maintainer: admin@opencv.org
Installed-Size: 5,160 kB
Provides: libopencv-dev, libopencv-calib3d-dev, libopencv-core-dev, libopencv-features2d-dev, libopencv-flann-dev, libopencv-highgui-dev, libopencv-imgcodecs-dev, libopencv-imgproc-dev, libopencv-ml-dev,
libopencv-objdetect-dev, libopencv-photo-dev, libopencv-stitching-dev, libopencv-ts-dev, libopencv-video-dev, libopencv-videoio-dev
Depends: libopencv (= 4.1.1-2-gd5a58aa75), libc6 (>= 2.14), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2)
Conflicts: libopencv-dev, libopencv-calib3d-dev, libopencv-core-dev, libopencv-features2d-dev, libopencv-flann-dev, libopencv-highgui-dev, libopencv-imgcodecs-dev, libopencv-imgproc-dev, libopencv-ml-dev, libopencv-objdetect-dev, libopencv-photo-dev, libopencv-stitching-dev, libopencv-ts-dev, libopencv-video-dev, libopencv-videoio-dev
Replaces: libopencv-dev, libopencv-calib3d-dev, libopencv-core-dev, libopencv-features2d-dev, libopencv-flann-dev, libopencv-highgui-dev, libopencv-imgcodecs-dev, libopencv-imgproc-dev, libopencv-ml-dev,
libopencv-objdetect-dev, libopencv-photo-dev, libopencv-stitching-dev, libopencv-ts-dev, libopencv-video-dev, libopencv-videoio-dev
Homepage: http://opencv.org
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Development files for Open Source Computer Vision Library

[psyhtest]

I suspect that OpenCV is not configured correctly in the first place in package/lib-openvino/scripts.linux/install.sh:

  -DENABLE_OPENCV=ON \
  -DOpenCV_DIR="${CK_ENV_LIB_OPENCV}" \

[psyhtest]

That's why system paths get used:

dvdt@7f9ed6e7346a:~/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/obj$ grep OpenCV * -R
CMakeCache.txt://enables OpenCV
CMakeCache.txt://The directory containing a CMake configuration file for OpenCV.
CMakeCache.txt:OpenCV_DIR:PATH=/usr/share/OpenCV
CMakeFiles/Makefile.cmake:  "/usr/share/OpenCV/OpenCVConfig-version.cmake"
CMakeFiles/Makefile.cmake:  "/usr/share/OpenCV/OpenCVConfig.cmake"
CMakeFiles/Makefile.cmake:  "/usr/share/OpenCV/OpenCVModules-release.cmake"
CMakeFiles/Makefile.cmake:  "/usr/share/OpenCV/OpenCVModules.cmake"
InferenceEngineDeveloperPackageConfig.cmake:# inherit OpenCV from main IE project
InferenceEngineDeveloperPackageConfig.cmake:load_cache("${cache_path}" READ_WITH_PREFIX "" OpenCV_DIR)
InferenceEngineDeveloperPackageConfig.cmake:find_package(OpenCV COMPONENTS imgcodecs)

[psyhtest]

dvdt@7f9ed6e7346a:~/CK_REPOS/ck-openvino/package/lib-openvino$ git diff
diff --git a/package/lib-openvino/scripts.linux/install.sh b/package/lib-openvino/scripts.linux/install.sh
index 1ff18b4..e961d8d 100644
--- a/package/lib-openvino/scripts.linux/install.sh
+++ b/package/lib-openvino/scripts.linux/install.sh
@@ -57,7 +57,7 @@ ${CK_ENV_TOOL_CMAKE_BIN}/cmake \
   -DENABLE_GNA=OFF \
   -DENABLE_CLDNN=OFF \
   -DENABLE_OPENCV=ON \
-  -DOpenCV_DIR="${CK_ENV_LIB_OPENCV}" \
+  -DOpenCV_DIR="${CK_ENV_LIB_OPENCV}/share/OpenCV" \
   -DENABLE_PYTHON=ON \
   -DPYTHON_EXECUTABLE=${CK_ENV_COMPILER_PYTHON_FILE} \
   -DPYTHON_INCLUDE_DIR=$(${CK_ENV_COMPILER_PYTHON_FILE} -c "from distutils.sysconfig import get_config_var; print('{}'.format(get_config_var('INCLUDEPY')))") \

[psyhtest]

With the above change, ie_api.so gets builds without any errors:

dvdt@7f9ed6e7346a:~/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64$ find . -name ie_api.so
./dldt/inference-engine/bin/intel64/Release/lib/python_api/python3.6/openvino/inference_engine/ie_api.so

However, when running calibration, it's not found:

######################################################################################
Running calibration ...

LD_LIBRARY_PATH=/home/dvdt/CK_TOOLS/lib-opencv-3.4.3-gcc-7.5.0-linux-64/install/lib::/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/lib/ ; /usr/bin/python3 /home/dvdt/CK_TO
OLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/calibrate.py -c /home/dvdt/CK_TOOLS/model-openvino-converted-from-tf-lib.openvino-pre-release-re
snet50/config.yml -M /home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/model-optimizer -C /home/dvdt/CK_TOOLS/model-openvino-converted-from-tf-lib.openvino-pre-release-re
snet50 -e /home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/lib/ --output_dir /home/dvdt/CK_TOOLS/model-openvino-converted-from-tf-lib.openvino-pre-release-resnet50
Traceback (most recent call last):
  File "/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/calibrate.py", line 17, in <module>
    import openvino.tools.calibration as calibration
  File "/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/openvino/tools/calibration/__init__.py", line 16, in <module>
    from .aggregated_statistics import AggregatedStatistics
  File "/home/dvdt/CK_TOOLS/lib-openvino-gcc-7.5.0-pre-release-linux-64/dldt/inference-engine/tools/calibration_tool/openvino/tools/calibration/aggregated_statistics.py", line 19, in <modu$
e>
    import openvino.inference_engine as ie
ModuleNotFoundError: No module named 'openvino.inference_engine'
Error: Calibration failed!

Should it be added to PYTHONPATH of lib.openvino?

[psyhtest]

Finally, we can build and validate this model (thanks @G4V!) e.g. ctuning/mlperf-inference-v0.5.openvino:

$ docker pull ctuning/mlperf-inference-v0.5.openvino:ubuntu-18.04
$ docker run ctuning/mlperf-inference-v0.5.openvino:ubuntu-18.04

so this can be closed.