tensorboardX is incompatible with protobuf 4.21.0

[daitakahashi]

This is originally found when our CI server installed protobuf 4.12.0, which was released today, as a dependency of the tensorboardX.

$ pip3 install tensorboardX  # installs the newest protobuf as a dependency
$ python3
Python 3.8.13 (default, Mar 17 2022, 16:53:17) 
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorboardX import SummaryWriter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/__init__.py", line 5, in <module>
    from .torchvis import TorchVis
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/torchvis.py", line 11, in <module>
    from .writer import SummaryWriter
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/writer.py", line 17, in <module>
    from .comet_utils import CometLogger
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/comet_utils.py", line 7, in <module>
    from .summary import _clean_tag
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/summary.py", line 13, in <module>
    from .proto.summary_pb2 import Summary
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/proto/summary_pb2.py", line 16, in <module>
    from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/proto/tensor_pb2.py", line 16, in <module>
    from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
>>> 

Environment

$ pip3 list | grep -E "torch|proto|tensor"
protobuf                      4.21.0
tensorboard                   2.7.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.0
tensorboardX                  2.5
torch                         1.10.1
$ python3 --version
Python 3.8.13

Thank you very much in advance.

[pengxutao]

pip install -U protobuf==3.20.1 hope this helps you

[daitakahashi]

pip install -U protobuf==3.20.1

This may be a temporal option, but in reality, I don't want to add a non-trivial dependency in my project...

[lanpa]

@daitakahashi Hi, did you mean that protobuf 4.12.0 will be installed during pip install tensorboardX?

[daitakahashi]

Sorry, that was protobuf 4.21.0 (not 4.12.0; I fixed the title). Here are the complete commandlines and their outputs;

$ pip3 uninstall protobuf
Found existing installation: protobuf 4.21.0
Uninstalling protobuf-4.21.0:
  Would remove:
    /Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/google/_upb/_message.abi3.so
    /Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/google/protobuf/*
    /Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/protobuf-4.21.0.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled protobuf-4.21.0
$ pip3 uninstall tensorboardX
Found existing installation: tensorboardX 2.5
Uninstalling tensorboardX-2.5:
  Would remove:
    /Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX-2.5.dist-info/*
    /Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/tensorboardX/*
Proceed (Y/n)? y
  Successfully uninstalled tensorboardX-2.5
$ pip3 install tensorboardX
Defaulting to user installation because normal site-packages is not writeable
Collecting tensorboardX
  Using cached tensorboardX-2.5-py2.py3-none-any.whl (125 kB)
Requirement already satisfied: numpy in ./Library/Python/3.8/lib/python/site-packages (from tensorboardX) (1.22.3)
Requirement already satisfied: six in ./Library/Python/3.8/lib/python/site-packages (from tensorboardX) (1.16.0)
Collecting protobuf>=3.8.0
  Downloading protobuf-4.21.1-cp37-abi3-macosx_10_9_universal2.whl (483 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 483.1/483.1 KB 3.4 MB/s eta 0:00:00
Installing collected packages: protobuf, tensorboardX
Successfully installed protobuf-4.21.1 tensorboardX-2.5

[lanpa]

Got your point, will have it fixed this weekend. Thanks for the report.

[erikfrey]

+1, brax uses tensorboardX and we are seeing this problem too.

But also, thanks for tensorboardX, we find it quite useful :-)

[erikfrey]

If possible, please regenerate the files in tensorboardX/proto/*_pb2.py with a newer protoc rather than pinning tensorboardX to an older protobuf version. We won't be able to use tensorboardX if it requires an older protobuf.

[lanpa]

@erikfrey It seems that protobuf <= 3.20.1 is needed. What is your protobuf version requirement? https://github.com/tensorflow/tensorboard/issues/5703

[erikfrey]

Oh yes, I see. Thanks for pointing out that this is an issue upstream. It looks like it's being worked on, but I think you're right to downgrade for the time being.

There's probably no issue for me as the libs we rely on are all experiencing the same problem.

[lanpa]

@daitakahashi v2.5.1 should solve this problem. @erikfrey What is the minimal protobuf version requirement for brax ?

[erikfrey]

@lanpa I just tried and it's working for Brax:

  Running setup.py develop for brax
Successfully installed Pillow-9.1.1 absl-py-1.1.0 brax-0.0.13 chex-0.1.3 cloudpickle-2.1.0 cycler-0.11.0 dataclasses-0.6 dm-tree-0.1.7 flatbuffers-2.0 flax-0.5.0 fonttools-4.33.3 grpcio-1.46.3 gym-0.24.0 gym-notices-0.0.7 importlib-metadata-4.11.4 jax-0.3.13 jaxlib-0.3.10 kiwisolver-1.4.2 matplotlib-3.5.2 msgpack-1.0.4 numpy-1.22.4 opt_einsum-3.3.0 optax-0.1.2 packaging-21.3 protobuf-3.20.1 pyparsing-3.0.9 python-dateutil-2.8.2 pytinyrenderer-0.0.13 scipy-1.8.1 six-1.16.0 tensorboardX-2.5.1 toolz-0.11.2 trimesh-3.12.5 typing-extensions-4.2.0 zipp-3.8.0

So looks like I'm OK with 3.20.1 - thanks for checking. I hope that when tensorboard upstream fixes this protobuf compatibility issue, it allows you to undo this change.

[llchan]

tensorboard now supports protobuf 4, can we reopen this and upgrade/unpin?

[zcarrico-fn]

@llchan , protobuf versions in tensorboardx no longer have an upper limit so all that's needed now is a new release of Tensorboardx. We're also interested in this because tensorboardx is a Ray dependency, so we need the new release to be able to use Ray with packages that require protobuf > 4. @lanpa , thank you for all the great work on Tensorboardx! Do you have an estimate for when a new release (i.e., Tensorboardx2.7) will be available? Please let me know if there's anything I can do to help 🙂

[zcarrico-fn]

thank you for releasing 2.6.1, which enables using protobuf > 4!

[lanpa]

Hi @zcarrico-fn, does 2.6.1 work as expected? For the next release, any idea for the new features are welcome!

[zcarrico-fn]

@lanpa , yes, works great! Thank you and I will let you know if we have any new feature ideas :)