search

onnx / tensorflow-onnx

Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX
Apache License 2.0
2.33k stars 432 forks source link

Pinned protobuf version leads to dependency conflict with tensorflow>=2.13 and gRPC on M1/M2 #2261

Open Zahlii opened 1 year ago

Zahlii commented 1 year ago

tf2onnx>=1.15 pins protobuf~=3.20.2. Tensorflow >=2.13 requires tf2onnx >= 1.15 due to https://github.com/onnx/tensorflow-onnx/pull/2215

In order to use gRPC natively with M1/M2 chips, we need at least grpcio>=1.51.3, which in turn requires protobuf>=4.x

As a result, I can NOT use tensorflow in combination with gRPC on M1/m2, as long as I need tensorflow >= 2.13.

Could we simply unpin the protobuf version in setup.py?

See also https://github.com/onnx/tensorflow-onnx/issues/2243

fatcat-z commented 1 year ago

Could you please try to reinstall protobuf to a 4.x version after you install tf2onnx successfully? It might work as well.

IIRC, keeping 3.20.2 is for support tensorflow older version.

Zahlii commented 1 year ago

For my use case, all tests seem to run through (I slightly changed the order, I first install everything including tf2onnx 1.14, then run a update with --no-deps for tf2onnx). However, I am building a library that will be installed by others internally, and I'd rather not have to ask every colleague to manually upgrade tf2onnx after having installed the library.

Zahlii commented 1 year ago

Actually, it seems there is another issue, starting with 2.14 (?) of tensorflow. Flatbuffers changed their version format from 2.xxx to 22.xxx, and tf2onnx seems to be incompatible with newer versions.

186.9 The conflict is caused by: 186.9 tensorflow 2.14.0 depends on flatbuffers>=23.5.26 186.9 tf2onnx 1.14.0 depends on flatbuffers<3.0 and >=1.12

Zahlii commented 1 year ago

197.7 The conflict is caused by: 197.7 tensorflow 2.13.0 depends on flatbuffers>=23.1.21 197.7 tf2onnx 1.14.0 depends on flatbuffers<3.0 and >=1.12

Ganga-Yadav commented 10 months ago

Could you please try to reinstall protobuf to a 4.x version after you install tf2onnx successfully? It might work as well.

IIRC, keeping 3.20.2 is for support tensorflow older version.

i used : Name: protobuf Version: 4.25.1 with tensorflow Version: 2.13 but it didn't work ..

fatcat-z commented 10 months ago

197.7 The conflict is caused by: 197.7 tensorflow 2.13.0 depends on flatbuffers>=23.1.21 197.7 tf2onnx 1.14.0 depends on flatbuffers<3.0 and >=1.12

Starts from tf2onnx 1.15.0, flatbuffers is not required to be <3.0.

fatcat-z commented 10 months ago

Could you please try to reinstall protobuf to a 4.x version after you install tf2onnx successfully? It might work as well. IIRC, keeping 3.20.2 is for support tensorflow older version.

i used : Name: protobuf Version: 4.25.1 with tensorflow Version: 2.13 but it didn't work ..

Could you please share more details about your scenario which doesn't work?

hmc-cs-mdrissi commented 9 months ago 
IIRC, keeping 3.20.2 is for support tensorflow older version.

Older tf versions already have constraint blocking protobuf<4. Adding constraint for older version of another package mostly introduces dependency conflicts and pushes towards vendoring the library/re-packaging it entirely. Especially as pip ecosystem lacks a good mechanism for overrides so I'd also find it helpful if constraint is dropped.

harshad16 commented 6 months ago

I agree with @hmc-cs-mdrissi insight, the pinned protobuf is causing issue with other python modules, which requires protobuf > 4 . Can the pinned version be dropped form tf2onnx?

fatcat-z commented 6 months ago

I agree with @hmc-cs-mdrissi insight, the pinned protobuf is causing issue with other python modules, which requires protobuf > 4 . Can the pinned version be dropped form tf2onnx?

Thanks for your comments..

At this moment, tf2onnx is still designed to support older tf versions as many as possible and some of them could not work on a version above 3.20.x. In addition, couple of dependencies of tf2onnx also required the protobuf version ~= 3.20.0.