search

mavlink / MAVSDK-Python

MAVSDK client for Python.
https://mavsdk.mavlink.io
BSD 3-Clause "New" or "Revised" License
313 stars 219 forks source link

dependency confict between grpcio and protobuf #547

Open mnumanuyar opened 1 year ago

mnumanuyar commented 1 year ago

I get the fallowing error when i try to install rewurments.txt as mentioned in readme.md:

ERROR: grpcio-tools 1.51.1 has requirement protobuf<5.0dev,>=4.21.6, but you'll have protobuf 3.20.1 which is incompatible.

i suspect this is because current dependencies are : protobuf<=3.20.1 grpcio>=1.50.0 but grpc requires protobuf>=4.21.3 since v1.49.0

git checkout 1.4.1 solved this problem for me.

However this problem seems to be here for couple of months and i am not familiar with this project, so maybe i am missing something

do we need to rollback grpc version? or do wee need to update our protobuf version?

JonasVautherin commented 1 year ago

Can you try updating protobuf?

mnumanuyar commented 1 year ago

i changed related lines in requirements.txt and proto/pb_plugins/requirements.txt as protobuf>=4.21.3,<5.0dev (copy paste from grpc repo). However I get : 

ERROR: protoc-gen-mavsdk 1.1.1 has requirement protobuf<=3.20.1,>=3.13, but you'll have protobuf 4.21.12 which is incompatible.

I couldn't find a reference for requirement of that repo unfortunately.

mnumanuyar commented 1 year ago

fyi: I tried my luck with ./other/tools/run_protoc.sh anyway and I get : 

[+] Generating plugins from 
 -> [+] Generated protobuf and gRPC bindings for action
Traceback (most recent call last):
  File "/home/numan/repos/workbench/mavsdkpython_git_dependencybug/venv/bin/protoc-gen-mavsdk", line 5, in <module>
    from protoc_gen_mavsdk.__main__ import main
  File "/home/numan/repos/workbench/mavsdkpython_git_dependencybug/venv/lib/python3.8/site-packages/protoc_gen_mavsdk/__init__.py", line 5, in <module>
    from .autogen import AutoGen
  File "/home/numan/repos/workbench/mavsdkpython_git_dependencybug/venv/lib/python3.8/site-packages/protoc_gen_mavsdk/autogen.py", line 15, in <module>
    import protoc_gen_mavsdk.mavsdk_options_pb2
  File "/home/numan/repos/workbench/mavsdkpython_git_dependencybug/venv/lib/python3.8/site-packages/protoc_gen_mavsdk/mavsdk_options_pb2.py", line 33, in <module>
    _descriptor.EnumValueDescriptor(
  File "/home/numan/repos/workbench/mavsdkpython_git_dependencybug/venv/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 755, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
--custom_out: protoc-gen-custom: Plugin failed with status code 1.
JonasVautherin commented 1 year ago

I couldn't find a reference for requirement of that repo unfortunately.

Could it be coming from here? I guess you installed protoc-gen-mavsdk from pip?

mnumanuyar commented 1 year ago

sry i meant i couldnt find the requirements.txt file of protoc-gen-mavsdk .

Also; yes i download from pip as described in readme.md. so probably this. However git links in that page points to MAVSDK-Proto repo. I assume it is generated and published from this. I couldnt find why protoc-gen-mavsdk needs protobuf<=3.20.1 in MAVSDK-Proto repo though

mnumanuyar commented 1 year ago

ok, I found it, it is in pb_plugins folder I will try to change and update it later

mnumanuyar commented 1 year ago

I just realised that is the same repo with our submodule. But then isnt it redundant to both using it as submodule and installing it via pip3 install -r requirements.txt -r requirements-dev.txt.

And also which one is used?

mnumanuyar commented 1 year ago

removing protoc-gen-mavsdk>=1.1.1 solves it, but i guess it is required for some other things.

JonasVautherin commented 1 year ago

And also which one is used?

The one which is in your path :innocent:

isnt it redundant to both using it as submodule and installing it

I suppose it could be in another repo, but... that would make one more repo :sweat_smile:

mnumanuyar commented 1 year ago

I tried checking out main branch in proto, bumping versions, then installing it, and then reverting it to back where it was. pip show showed my version, but trying ./other/tools/run_protoc.sh resulted in same error

uninstalling protoc-gen-mavsdk and trying again worked (I did not extensively test it but there were no errors)

I think i am out of my depth here :sweat_smile: I hope i was able to help 😄

julianoes commented 1 year ago

I'm pretty sure I tried to "fix" this by trying to use protobuf 3.20.1 everywhere. I was hoping that would work for now but that might not be true.

And then with v2 we should probably go to latest protobuf 4.20...

mnumanuyar commented 1 year ago

well, protobuf>=4.21.3 to be precise, as grpc requires this, but yes.

mnumanuyar commented 1 year ago

fyi: to build latest in docker i had to add pip3 install grpcio-tools==1.48.2 grpcio==1.51.1

julianoes commented 1 year ago

Do you mind making a pull request to fix the dockerfiles?

mnumanuyar commented 1 year ago

i will try to add pip3 install grpcio-tools==1.48.2 grpcio==1.51.1 in https://github.com/mavlink/MAVSDK/blob/main/docker/Dockerfile-Ubuntu-20.04 (and also other ubuntu versions?) and test it tonight and mention this issue in pr

mnumanuyar commented 1 year ago

ok so here is the thing:

but other than that no problem. pip freeze shows

aiogrpc==1.8
certifi==2019.11.28
chardet==3.0.4
dbus-python==1.2.16
future==0.18.2
grpcio==1.53.0
grpcio-tools==1.53.0
idna==2.8
Jinja2==3.1.2
MarkupSafe==2.1.2
-e git+https://github.com/mavlink/MAVSDK-Python@2345933c5f14eb5da37569eb881fb9e86b68c4f4#egg=mavsdk
protobuf==3.20.1
protoc-gen-mavsdk==1.1.1
PyGObject==3.36.0
python-apt==2.0.1+ubuntu0.20.4.1
requests==2.22.0
requests-unixsocket==0.2.0
six==1.14.0
urllib3==1.25.8

Previosly my dockers related to mavsdk-python were failing, but mabe it is fixed? Or maybe it is something related to caches? I will try to continue to repro the issue, if I found out anything i will write