search

onnx / onnx-tensorflow

Tensorflow Backend for ONNX
Other
1.29k stars 296 forks source link

SoftmaxCrossEntropyLoss is not implemented #813

Open roberthajdu92 opened 4 years ago

roberthajdu92 commented 4 years ago

Describe the bug

Trying to convert an NLP model made with ORT solution to TF.

Log:

2020-11-19 17:44:18.001643: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-19 17:44:24,224 - onnx-tf - INFO - Start converting onnx pb to tf pb:
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 1482361365
2020-11-19 17:44:52.178513: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-11-19 17:44:52.267895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0001:00:00.0 name: Tesla V100-PCIE-16GB computeCapability: 7.0
coreClock: 1.38GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2020-11-19 17:44:52.267941: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-19 17:44:52.269430: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-19 17:44:52.270828: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-19 17:44:52.271079: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-19 17:44:52.272535: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-19 17:44:52.273214: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-19 17:44:52.276479: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-19 17:44:52.278152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-19 17:44:52.278505: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-19 17:44:52.284917: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2593990000 Hz
2020-11-19 17:44:52.285669: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56182f180920 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-19 17:44:52.285694: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-11-19 17:44:52.421515: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56182f1a4430 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-11-19 17:44:52.421557: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2020-11-19 17:44:52.422524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0001:00:00.0 name: Tesla V100-PCIE-16GB computeCapability: 7.0
coreClock: 1.38GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2020-11-19 17:44:52.422570: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-19 17:44:52.422602: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-19 17:44:52.422624: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-19 17:44:52.422645: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-19 17:44:52.422665: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-19 17:44:52.422686: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-19 17:44:52.422706: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-19 17:44:52.424293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-19 17:44:52.424333: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-19 17:44:52.874789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-19 17:44:52.874836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-11-19 17:44:52.874848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-11-19 17:44:52.876678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14763 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0001:00:00.0, compute capability: 7.0)
WARNING:tensorflow:Large unrolled loop detected. Did you mean to use a TF loop? The following ops were created after iteration 3002: (<tf.Operation 'onnx_tf_prefix_Sub_3000' type=Sub>,)
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/common_errors.md#warning-large-unrolled-loop-detected
Location:
  File "/anaconda/envs/azureml_py36/bin/onnx-tf", line 33, in <module>
    sys.exit(load_entry_point('onnx-tf', 'console_scripts', 'onnx-tf')())

  File "onnx-tensorflow/onnx_tf/cli.py", line 20, in main
    return onnx_tf.converter.main(args[1:])

  File "onnx-tensorflow/onnx_tf/converter.py", line 20, in main
    convert(**{k: v for k, v in vars(args).items() if v is not None})

  File "onnx-tensorflow/onnx_tf/converter.py", line 135, in convert
    tf_rep.export_graph(outdir)

  File "onnx-tensorflow/onnx_tf/backend_rep.py", line 116, in export_graph
    **self.signatures))

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 1167, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 1073, in _get_concrete_function_garbage_collected
    self._initialize(args, kwargs, add_initializers_to=initializers)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3735, in bound_method_wrapper
    return wrapped_fn(*args, **kwargs)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 969, in wrapper
    user_requested=True,

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 596, in converted_call
    result = converted_f(*effective_args, **kwargs)

  File "/tmp/tmpy0dcdt2s.py", line 30, in tf____call__
    ag__.for_stmt(ag__.ld(self).graph_def.node, None, loop_body, get_state, set_state, (), {'iterate_names': 'node'})

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 368, in for_stmt
    _py_for_stmt(iter_, extra_test, body, None, None)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 397, in _py_for_stmt
    body(target)

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 384, in protected_body
    after_iteration()

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 834, in after_iteration
    did_warn = self._verify_inefficient_unroll()

  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 817, in _verify_inefficient_unroll
    '', self.iterations, new_ops, '\n'.join(traceback.format_stack()))

Traceback (most recent call last):
  File "/anaconda/envs/azureml_py36/bin/onnx-tf", line 33, in <module>
    sys.exit(load_entry_point('onnx-tf', 'console_scripts', 'onnx-tf')())
  File "onnx-tensorflow/onnx_tf/cli.py", line 20, in main
    return onnx_tf.converter.main(args[1:])
  File "onnx-tensorflow/onnx_tf/converter.py", line 20, in main
    convert(**{k: v for k, v in vars(args).items() if v is not None})
  File "onnx-tensorflow/onnx_tf/converter.py", line 135, in convert
    tf_rep.export_graph(outdir)
  File "onnx-tensorflow/onnx_tf/backend_rep.py", line 116, in export_graph
    **self.signatures))
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 1167, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 1073, in _get_concrete_function_garbage_collected
    self._initialize(args, kwargs, add_initializers_to=initializers)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3735, in bound_method_wrapper
    return wrapped_fn(*args, **kwargs)
  File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
onnx.backend.test.runner.BackendIsNotSupposedToImplementIt: in user code:

    onnx-tensorflow/onnx_tf/backend_tf_module.py:98 __call__  *
        output_ops = self.backend._onnx_node_to_tensorflow_op(onnx_node,
    onnx-tensorflow/onnx_tf/backend.py:291 _onnx_node_to_tensorflow_op  *
        raise BackendIsNotSupposedToImplementIt("{} is not implemented.".format(

    BackendIsNotSupposedToImplementIt: SoftmaxCrossEntropyLoss is not implemented.

To Reproduce

onnx-tf convert -i ../PyTorch/final_bert.onnx -o output.pb

ONNX model file

Cannot share it unfortunately.

Python, ONNX, ONNX-TF, Tensorflow version

This section can be obtained by running get_version.py from util folder.

Additional context

Model opset: 12

chinhuang007 commented 3 years ago

Unfortunately the training ops are not supported in onnx-tf. Contribution is very much welcome!