Open cjvolzka opened 6 months ago
Support bfloat16 and float16 scales. Support float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz quantized tensors:
I'll note that this fails with model using bfloat16 and latest versions of tf2onnx/onnx/onnxruntime. Error message looks like,
File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tfonnx.py", line 459, in process_tf_graph
main_g, subgraphs = graphs_from_tf(tf_graph, input_names, output_names, shape_override, const_node_values,
File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tfonnx.py", line 474, in graphs_from_tf
ordered_func = resolve_functions(tf_graph)
File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tf_loader.py", line 784, in resolve_functions
_, _, _, _, _, functions = tflist_to_onnx(tf_graph, {})
File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tf_utils.py", line 443, in tflist_to_onnx
onnx_tensor = tf_to_onnx_tensor(value, name=port_name(node.name))
File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tf_utils.py", line 65, in tf_to_onnx_tensor
return numpy_helper.from_array(np_data, name=name)
File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/onnx/numpy_helper.py", line 324, in from_array
raise RuntimeError(
RuntimeError: Numpy data type not understood yet: bfloat16
Unsure if fix is better then in tf2onnx side in tf_to_onnx_tensor
or in onnx side in numpy_helper
. I see a comment
# NumPy doesn't have BFLOAT16.
in onnx and this issue is still open so I think fix makes more sense in tf2onnx and the assumption that tensorflow tensor can be converted to numpy and then to onnx is not true for bfloat16.
We are releasing ONNX 1.16.0. A release branch is created (https://github.com/onnx/onnx/tree/rel-1.16.0). The planned release date is March 25, 2024. Release candidates are also available from TestPyPI:
pip install -i https://test.pypi.org/simple/ --pre onnx
It is important to integrate ONNX release branch ASAP so that any issues and incompatibilities can be detected and resolved before the ONNX release.
Key updates:
stash_type
attribute to GroupNormalizationIn case a bug in ONNX is detected during integration of ONNX 1.16.0, please open a ONNX Bug Report and tag ONNX Release Manager @cjvolzka so that the bug is fixed in the ONNX release branch.