v1.2.14 on Linux Mint 21 refuses to launch; likely a dependency conflict

kerastinell commented 1 year ago

TL;DR: Warpinator v1.2.14 works great on Linux Mint 21 until I install Meshtastic CLI from pip. I wasn't sure about where to open this Issue and I started here.

OS: Linux Mint 21, almost clean install except:

~# apt install python3-pip
~$ pip3 install -U pip
( then add "$HOME/.local/bin" to $PATH )

Warpinator: installed with apt:

~# apt install warpinator
...........
~$ dpkg --list | fgrep -i warpinator
ii  warpinator                                    1.2.14+vanessa                             all          Allows simple local network file sharing.

Under these conditions Warpinator works fine.

Warpinator refuses to start after I install Meshtastic CLI (github). Installation log:

~$ pip install -U meshtastic
pip install -U meshtastic
Defaulting to user installation because normal site-packages is not writeable
Collecting meshtastic
  Using cached meshtastic-1.3.37-py3-none-any.whl (76 kB)
Collecting pyqrcode>=1.2.1
  Using cached PyQRCode-1.2.1-py3-none-any.whl
Collecting timeago>=1.0.15
  Using cached timeago-1.0.16-py3-none-any.whl (29 kB)
Collecting tabulate>=0.8.9
  Using cached tabulate-0.8.10-py3-none-any.whl (29 kB)
Collecting pygatt>=4.0.5
  Using cached pygatt-4.0.5-py3-none-any.whl
Collecting protobuf>=3.13.0
  Using cached protobuf-4.21.7-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
Collecting pyserial>=3.4
  Using cached pyserial-3.5-py2.py3-none-any.whl (90 kB)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from meshtastic) (5.4.1)
Collecting pypubsub>=4.0.3
  Using cached Pypubsub-4.0.3-py3-none-any.whl (61 kB)
Requirement already satisfied: pexpect>=4.6.0 in /usr/lib/python3/dist-packages (from meshtastic) (4.8.0)
Collecting dotmap>=1.3.14
  Using cached dotmap-1.3.30-py3-none-any.whl (11 kB)
Collecting enum-compat
  Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB)
Installing collected packages: timeago, pyserial, pyqrcode, enum-compat, dotmap, tabulate, pypubsub, pygatt, protobuf, meshtastic
Successfully installed dotmap-1.3.30 enum-compat-0.0.3 meshtastic-1.3.37 protobuf-4.21.7 pygatt-4.0.5 pypubsub-4.0.3 pyqrcode-1.2.1 pyserial-3.5 tabulate-0.8.10 timeago-1.0.16

Warpinator log:

~$ warpinator
Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

There seems to be some dependency compatibility issue with the protobuf package. Meshtastic CLI works fine regardless of the presence of Warpinator in my system. Warpinator in its turn refuses to start when Meshtastic CLI is installed.

Using the 2nd suggestion in the error log makes Warpinator work again. I didn't try the 1st suggestion, though.

kerastinell commented 1 year ago

I opened an Issue in the Meshtastic CLI repository as well: https://github.com/meshtastic/Meshtastic-python/issues/378

mtwebster commented 1 year ago

This is a bit of a rabbit hole.. I'm certain there's nothing wrong with meshtastic.

Short history:

Mint 21/Ubuntu 22.04's shipped version of python3-grpcio is broken - warpinator starts, freezes as soon as the server starts. See #127, #138.
As a workaround, we started including python-grpc 1.46.3 for Mint 21 packages.
Soon it was discovered all python-grpc versions > 1.44 are using too much cpu at idle (#137).
So we backed down to 1.44 (8fd5f86230bc0aee5b)

Just last week there was a security flaw detected in protobuf: https://groups.google.com/g/grpc-io/c/msTHLzIA9n0/m/ClQKCXu9BAAJ?utm_medium=email&utm_source=footer

I'm pretty sure your specific issue is that Warpinator's bundled grpcio is too old for the protobuf version you have. Setting that environment variable should temporarily resolve it (PROTOCOL_BUFFERS...)

I'm hoping to get a resolution to the CPU issue so I can actually use the newer versions of grpc that will utilize the 'fixed' protobuf versions. You can also try:

Install the latest grpcio from pip (1.49 or something) along with grpcio-tools.
Get the warpinator source, run generate-protobuf
cp src/warp_pb2*.py /usr/libexec/warpinator (replace the old protobuf files)
rm -rf /usr/libexec/warpinator/grpc (remove the bundled grpc so it uses pip's)

Or, you could just use the flatpak version. It currently suffers from the cpu issue - if you have a surplus of cores it's not a huge deal - but is fine otherwise.

Sorry for the length..

mtwebster commented 1 year ago

It looks like it's enough to just regenerate the protobuf files.

warp-proto.tar.gz

Try copying those into /usr/libexec/warpinator

kerastinell commented 1 year ago

I installed grpcio and grpcio-tools from pip and re-generated protobuf files. After I copied them into /usr/libexec/warpinator Warpinator started working again.

While this indeed worked I doubt I need to close this issue because a proper solution should be implemented. @mtwebster , thank you for your kind help!

kerastinell commented 1 year ago

Perhaps it's possible to implement some sort of a post-install hook for a package manager that will check for the presence of the newer grpc stuff in the system and re-generate protobufs? I don't have enough expertise on such matters, just wanted to contribute my suggestion.

mtwebster commented 1 year ago

Well, it's usually not a good idea to mix pip packages with system ones.

You're invariably going to run into issues such as when you install module A, which requires a newer version of module B (which is currently provided by the system). You upgrade package B using pip, and now you've broken system module C, which relied on an older version of B.

You can generally get away with it for obscure, isolated packages, but I would try to avoid it as much as possible.

I'll probably start bundling grpc and protobuf for all to avoid this specific problem in the future. They're critical for warpinator, but also widely used elsewhere. We bundle zeroconf already due to breaking changes from version to version.

You're best off creating a separate environment to run in when you must use pip packages.

sajalbose commented 1 year ago

This issue is still present in the latest linux mint,file --> vera/backport amd64 warpinator all 1.4.4+victoria [3,591 kB]

Though alternatively samba or ssh works for me,I would try to see rebuilding works or not.

And yes the error message

`Descriptors cannot not be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are:

Downgrade the protobuf package to 3.20.x or lower.
Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates `

Its already mentioned above by @kerastinell

tonywhelan commented 1 year ago

It looks like it's enough to just regenerate the protobuf files.

warp-proto.tar.gz

Try copying those into /usr/libexec/warpinator

I had the same warpinator error in LM 21.1 and LM 21.2 beta but don't know which of the extra deb packages I installed was the cause of it.

As suggested by mtwebster I tried copying the two files from warp-proto.tar.gz into /usr/libexec/warpinator and that fixed warpinator.

tonywhelan commented 1 year ago

I did some more testing on a virtual install of LM21.2 beta, and the package that broke warpinator was "overgrive" which installs a number of packages including: javascript-common libexpat1-dev libjs-jquery libjs-sphinxdoc libjs-underscore libpython3-dev libpython3.10-dev python3-dev python3-pip python3-wheel python3.10-dev zlib1g-dev and then cachetools-5.3.1 certifi-2023.5.7 charset-normalizer-3.1.0 google-api-core-2.11.1 google-api-python-client-2.91.0 google-auth-2.21.0 google-auth-httplib2-0.1.0 google-auth-oauthlib-1.0.0 googleapis-common-protos-1.59.1 httplib2-0.22.0 idna-3.4 oauthlib-3.2.2 protobuf-4.23.3 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyparsing-3.1.0 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 uritemplate-4.1.1 urllib3-1.26.16

I guess its the protobuf package that caused the problem. Purging the overgrive package does not restore warpinator functionality, so I'll stick with the workaround of copying over two updated py files as suggested above by mtwebster

linuxmint / warpinator

v1.2.14 on Linux Mint 21 refuses to launch; likely a dependency conflict #145