please regenerate protobuf _pb2.py files using recent protoc, and publish new release on PyPI

[SomberNight]

Unfortunately Google made breaking changes in protobuf (see e.g. here), and hence the current _pb2.py generated files in latest release/master of keepkeylib cannot be parsed with new versions of protobuf (4.x).

AFAICT protoc >=3.19 generates _pb2 files in the new format, while older protoc generates the old format. AFAICT

• protobuf 3.20.x is able to parse both the old and the new format,
• protobuf <3.20 can only parse the old format,
• protobuf >=3.21 can only parse the new format

user@user-VirtualBox:~/wspace/tmp$ git clone https://github.com/keepkey/python-keepkey
Cloning into 'python-keepkey'...
remote: Enumerating objects: 8669, done.
remote: Counting objects: 100% (54/54), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 8669 (delta 31), reused 33 (delta 17), pack-reused 8615
Receiving objects: 100% (8669/8669), 56.08 MiB | 606.00 KiB/s, done.
Resolving deltas: 100% (3780/3780), done.

user@user-VirtualBox:~/wspace/tmp$ python3 -m venv env
user@user-VirtualBox:~/wspace/tmp$ source env/bin/activate

(env) user@user-VirtualBox:~/wspace/tmp$ cd python-keepkey/
(env) user@user-VirtualBox:~/wspace/tmp/python-keepkey$ git log -n1
commit 7bddc11d7ee8208161d8b428ad5b6760dd450a98 (HEAD -> master, origin/master, origin/HEAD)
Merge: 4054710 866fc63
Author: markrypt0 <91701444+markrypt0@users.noreply.github.com>
Date:   Sun Aug 7 07:34:19 2022 -0600

    Merge pull request #145 from keepkey/xpub-support

    Xpub support

(env) user@user-VirtualBox:~/wspace/tmp/python-keepkey$ pip install -e .
Obtaining file:///home/user/wspace/tmp/python-keepkey
  Preparing metadata (setup.py) ... done
Collecting ecdsa>=0.9
  Using cached ecdsa-0.18.0-py2.py3-none-any.whl (142 kB)
Collecting hidapi>=0.7.99.post15
  Using cached hidapi-0.12.0.post2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (892 kB)
Collecting libusb1>=1.6
  Using cached libusb1-3.0.0-py3-none-any.whl (62 kB)
Collecting mnemonic>=0.8
  Using cached mnemonic-0.20-py3-none-any.whl (62 kB)
Collecting protobuf>=3.0.0
  Using cached protobuf-4.21.4-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
Collecting six>=1.9.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools>=19.0 in /home/user/wspace/tmp/env/lib/python3.10/site-packages (from hidapi>=0.7.99.post15->keepkey==7.0.3) (59.6.0)
Installing collected packages: libusb1, six, protobuf, mnemonic, hidapi, ecdsa, keepkey
  Running setup.py develop for keepkey
Successfully installed ecdsa-0.18.0 hidapi-0.12.0.post2 keepkey-7.0.3 libusb1-3.0.0 mnemonic-0.20 protobuf-4.21.4 six-1.16.0

(env) user@user-VirtualBox:~/wspace/tmp/python-keepkey$ python ./helloworld.py 
Traceback (most recent call last):
  File "/home/user/wspace/tmp/python-keepkey/./helloworld.py", line 4, in <module>
    from keepkeylib.client import KeepKeyClient
  File "/home/user/wspace/tmp/python-keepkey/keepkeylib/client.py", line 37, in <module>
    from . import mapping
  File "/home/user/wspace/tmp/python-keepkey/keepkeylib/mapping.py", line 1, in <module>
    from . import messages_pb2 as proto
  File "/home/user/wspace/tmp/python-keepkey/keepkeylib/messages_pb2.py", line 17, in <module>
    from . import types_pb2 as types__pb2
  File "/home/user/wspace/tmp/python-keepkey/keepkeylib/types_pb2.py", line 34, in <module>
    _descriptor.EnumValueDescriptor(
  File "/home/user/wspace/tmp/env/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 755, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
(env) user@user-VirtualBox:~/wspace/tmp/python-keepkey$ 

related: https://github.com/digitalbitbox/bitbox02-firmware/pull/951

---

I would suggest

1. using latest protoc to re-generate the _pb2.py files

   $ mkdir -p /opt/protoc && \
     curl -L0 https://github.com/protocolbuffers/protobuf/releases/download/v21.2/protoc-21.2-linux-x86_64.zip -o /tmp/protoc-21.2-linux-x86_64.zip && \
     unzip /tmp/protoc-21.2-linux-x86_64.zip -d /opt/protoc
   $ PATH="/opt/protoc/bin:$PATH"
   $ ./build_pb.sh

2. bumping required protobuf in setup.py to "protobuf>=3.20" https://github.com/keepkey/python-keepkey/blob/7bddc11d7ee8208161d8b428ad5b6760dd450a98/setup.py#L16
3. then releasing a new version of this library on PyPI
   • btw, I see there has been multiple git tags since the last PyPI release (6.3.1 in Dec 2019). Please also release there.

[SomberNight]

What is the state of this? A reply would be nice. :)

Note: for continued support of the keepkey plugin in Electrum, we need a new release of the library on PyPI with these changes. If you've lost the keys to the PyPI package, just release under a new name and state it in the README or similar.

[SomberNight]

Even if you don't release a new version on PyPI, could you regenerate the pb2 files in the git repo? I can open a PR for it if you want, but please say so here then.

Otherwise if this library is unmaintained, again, we might have to remove keepkey support from Electrum.

[MrNaif2018]

CC @markrypt0 @pastaghost (I see they are not in repository's watchers list, so they might miss the issue accidentally)

[pastaghost]

@SomberNight @MrNaif2018 - thanks for tagging me on this issue; we'll give this some attention right away.

[markrypt0]

There are several reasons this hasn't been updated. First is that it isn't a seamless upgrade in firmware and because it hasn't been a critical issue, nothing has been changed. Second, the current keepkey pypl account credentials are not available so this is a problem that needs to be fixed.

[MrNaif2018]

@markrypt0 As for the latter: I think it's not much of a problem, another pypi package can be created, or even it's possible to install from github I think it shouldn't be a breaking change anyway but an improvement in speed

[cculianu]

BUMP. Can you please just create a new pypi package with latest and with recompiled _pb2.py files? Electron Cash for BCH also would like this else we may drop keepkey support as well.

EDIT: And like @SomberNight said, you can create a new package under a different username on pypi. It's fine.

[MrNaif2018]

Technically worst case it is even possible to load from a github tag specified in requirements.txt, but indeed nothing stops from just publishing a package under different name

[zquestz]

I have a fork that has implemented these changes. Would love some extra eyes and testers.

https://github.com/zquestz/python-keepkey

[PiRK]

I have a fork that has implemented these changes. Would love some extra eyes and testers.

https://github.com/zquestz/python-keepkey

I don't have a keepkey device for testing this properly with Electrum ABC or Electron Cash, but I'm getting an error with a basic import

$ pip install https://github.com/zquestz/python-keepkey/archive/9e4af3c0e349eaaa395091b35a7202066fc820af.tar.gz
$ python
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keepkeylib
>>> from keepkeylib import client
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/client.py", line 37, in <module>
    from . import mapping
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/mapping.py", line 1, in <module>
    from . import messages_pb2 as proto
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/messages_pb2.py", line 25, in <module>
    import types_pb2 as types__pb2
ModuleNotFoundError: No module named 'types_pb2'

[zquestz]

I have a fork that has implemented these changes. Would love some extra eyes and testers. https://github.com/zquestz/python-keepkey

I don't have a keepkey device for testing this properly with Electrum ABC or Electron Cash, but I'm getting an error with a basic import

$ pip install https://github.com/zquestz/python-keepkey/archive/9e4af3c0e349eaaa395091b35a7202066fc820af.tar.gz
$ python
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keepkeylib
>>> from keepkeylib import client
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/client.py", line 37, in <module>
    from . import mapping
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/mapping.py", line 1, in <module>
    from . import messages_pb2 as proto
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/messages_pb2.py", line 25, in <module>
    import types_pb2 as types__pb2
ModuleNotFoundError: No module named 'types_pb2'

Will take a look at this shortly. =)

[zquestz]

I have a fork that has implemented these changes. Would love some extra eyes and testers. https://github.com/zquestz/python-keepkey

I don't have a keepkey device for testing this properly with Electrum ABC or Electron Cash, but I'm getting an error with a basic import

$ pip install https://github.com/zquestz/python-keepkey/archive/9e4af3c0e349eaaa395091b35a7202066fc820af.tar.gz
$ python
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keepkeylib
>>> from keepkeylib import client
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/client.py", line 37, in <module>
    from . import mapping
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/mapping.py", line 1, in <module>
    from . import messages_pb2 as proto
  File "/home/pierre/.local/lib/python3.10/site-packages/keepkeylib/messages_pb2.py", line 25, in <module>
    import types_pb2 as types__pb2
ModuleNotFoundError: No module named 'types_pb2'

Pushed a fix. Client now imports correctly, had some issues with Mayachain code and the way the imports were generated.

Python 3.12.5 (main, Aug  9 2024, 08:20:41) [GCC 14.2.1 20240805] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keepkeylib
>>> from keepkeylib import client
>>> client
<module 'keepkeylib.client' from '/home/quest/src/python-keepkey/keepkeylib/client.py'>

[PiRK]

Pushed a fix. Client now imports correctly, had some issues with Mayachain code and the way the imports were generated.

OK, my very basic unit test is passing now. Good enough for me, until we find an actual tester with access to a device. Thanks @zquestz!