Retrain SSDLite MobileDet detector for the Edge TPU (TF1) error with NotImplementedError

[taro-ishihara]

Description

I tried tutorial "Retrain SSDLite MobileDet detector for the Edge TPU (TF1)" and exec python3 object_detection/model_main.py and then an error has occurred with message "NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array.".

I have not changed anything from the sample, and I use just GoogleColab. Any ideas?

Click to expand!

### Issue Type Build/Install ### Operating System Mac OS ### Coral Device _No response_ ### Other Devices _No response_ ### Programming Language Python 3.7 ### Relevant Log Output ```shell Traceback (most recent call last): File "object_detection/model_main.py", line 108, in tf.app.run() File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run _run_main(main, args) File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "object_detection/model_main.py", line 104, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default input_fn, ModeKeys.TRAIN)) File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn self._call_input_fn(input_fn, mode)) File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 1116, in _call_input_fn return input_fn(**kwargs) File "/content/models/research/object_detection/inputs.py", line 764, in _train_input_fn params=params) File "/content/models/research/object_detection/inputs.py", line 907, in train_input reduce_to_frame_fn=reduce_to_frame_fn) File "/content/models/research/object_detection/builders/dataset_builder.py", line 252, in build input_reader_config) File "/content/models/research/object_detection/builders/dataset_builder.py", line 237, in dataset_map_fn fn_to_map, num_parallel_calls=num_parallel_calls) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/util/deprecation.py", line 324, in new_func return func(*args, **kwargs) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 1950, in map_with_legacy_function use_legacy_function=True)) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 3472, in __init__ use_legacy_function=use_legacy_function) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 2689, in __init__ self._function.add_to_graph(ops.get_default_graph()) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/function.py", line 545, in add_to_graph self._create_definition_if_needed() File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/function.py", line 377, in _create_definition_if_needed self._create_definition_if_needed_impl() File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/function.py", line 408, in _create_definition_if_needed_impl capture_resource_var_by_value=self._capture_resource_var_by_value) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/function.py", line 944, in func_graph_from_py_func outputs = func(*func_graph.inputs) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 2681, in wrapper_fn ret = _wrapper_helper(*args) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 2652, in _wrapper_helper ret = autograph.tf_convert(func, ag_ctx)(*nested_args) File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) NotImplementedError: in converted code: /content/models/research/object_detection/data_decoders/tf_example_decoder.py:580 decode default_groundtruth_weights) /tensorflow-1.15.2/python3.7/tensorflow_core/python/util/deprecation.py:507 new_func return func(*args, **kwargs) /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/control_flow_ops.py:1235 cond orig_res_f, res_f = context_f.BuildCondBranch(false_fn) /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/control_flow_ops.py:1061 BuildCondBranch original_result = fn() /content/models/research/object_detection/data_decoders/tf_example_decoder.py:573 default_groundtruth_weights dtype=tf.float32) /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2560 ones output = _constant_if_small(one, shape, dtype, name) /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2295 _constant_if_small if np.prod(shape) < 1000: <__array_function__ internals>:6 prod /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:3052 prod keepdims=keepdims, initial=initial, where=where) /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:86 _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py:736 __array__ " array.".format(self.name)) NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array. ```

[hjonnala]

Hello @taro-ishihara can you please try the colab with pycocotools installation (! pip install pycocotools==2.0.0). Thanks!

[taro-ishihara]

hi @jjonnala I have tried to implement your advice now, but the situation did not change anything. Because this line (! apt-get install protobuf-compiler) automatically install pycocotools 2.0.0. Thanks.

Then I tried downgrading and upgrading numpy.

• downgrade to 1.19.5 Error is changed. This is indeed an error from pycocotools, but it seems that numpy version has to be newer.

  RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
  Traceback (most recent call last):
  File "object_detection/model_main.py", line 25, in <module>
  from object_detection import model_lib
  File "/usr/local/lib/python3.7/dist-packages/object_detection/model_lib.py", line 29, in <module>
  from object_detection import eval_util
  File "/usr/local/lib/python3.7/dist-packages/object_detection/eval_util.py", line 35, in <module>
  from object_detection.metrics import coco_evaluation
  File "/usr/local/lib/python3.7/dist-packages/object_detection/metrics/coco_evaluation.py", line 25, in <module>
  from object_detection.metrics import coco_tools
  File "/usr/local/lib/python3.7/dist-packages/object_detection/metrics/coco_tools.py", line 51, in <module>
  from pycocotools import coco
  File "/usr/local/lib/python3.7/dist-packages/pycocotools/coco.py", line 52, in <module>
  from . import mask as maskUtils
  File "/usr/local/lib/python3.7/dist-packages/pycocotools/mask.py", line 3, in <module>
  import pycocotools._mask as _mask
  File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
  ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

• upgrade to 1.20.0(next to 1.19.5) and 1.21.5(latest) Nothing changed.

[taro-ishihara]

OK, finally I solved this issue. Numpy downgrading AND pycocotools installation is needed.

! pip install -U numpy==1.19.5
! pip install -U pycocotools==2.0.0

Thanks, @hjonnala

[google-coral-bot[bot]]

Are you satisfied with the resolution of your issue? Yes No