tensorflow / tensorboard

TensorFlow's Visualization Toolkit
Apache License 2.0
6.67k stars 1.65k forks source link

protoc compiler is too old for tensorflow 2.12 #6307

Open jzhoulon opened 1 year ago

jzhoulon commented 1 year ago

tensorflow 2.12 has upgrade its protoc compiler to 3.21.9 while tensorboard still keeps 3.19.6, this will make profiler data(xplane.xxx.pb) failed to load by tensorboard.
tensorboard proto: https://github.com/tensorflow/tensorboard/blob/56b38b75dfb993d0da6a241114bb475d072b5ab9/WORKSPACE#L138 tensorflow proto https://github.com/tensorflow/tensorflow/blob/e8473900214fb5508699ebeafd7ea5f3ef0c2fd1/tensorflow/workspace2.bzl#L463

The error messeage is as follows:

ackages/tensorboard_plugin_profile/protobuf/diagnostics_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/home/subratag/mc3/envs/23ww14/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 561, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower). 
ericdnielsen commented 1 year ago

I think the title should be 2.12 not 1.12, correct? I'll edit if that was a typo.

We've seen this before, and its due to the profiler plugin using an old protobuf compiler. TB can not update the protobuf compiler since we need to maintain backward compatibility with older version of TF. The profiler plugin was updated to a working version last week, can you check if you have that? Specifically you want to see: https://github.com/tensorflow/profiler/blob/master/plugin/setup.py#L26

jzhoulon commented 1 year ago

I think the title should be 2.12 not 1.12, correct? I'll edit if that was a typo.

We've seen this before, and its due to the profiler plugin using an old protobuf compiler. TB can not update the protobuf compiler since we need to maintain backward compatibility with older version of TF. The profiler plugin was updated to a working version last week, can you check if you have that? Specifically you want to see: https://github.com/tensorflow/profiler/blob/master/plugin/setup.py#L26

sorry for the typo, it is 2.12, i edit it. thanks

jzhoulon commented 1 year ago

@ericdnielsen I install the profiler plugin and pip install the protobuf==3.20.1, the protoc issue is disappeared, however the profiler data can't show in tensorboard, I have host.xplane.pb in log dir, however, it reports no event files

======================================================================
Processing event files... (this can take a few minutes)
======================================================================

No event files found within logdir data
ericdnielsen commented 1 year ago

At this point I think we should hand you off to the profiler team: https://github.com/tensorflow/profiler

dpinol commented 1 year ago

Hi, any plans to upgrade protobuf to >=3.20? It was release more than 1 year ago and some other libraries (eg. streamlit) require at least 3.20 thanks

ahmedetefy commented 1 year ago

Bumping this as well, we use protobuf 4.21.3 and we keep running into the same issue. Are there plans of bumping the protoc compiler for TB?