juju-backup-all failed on new release when protobuf lib is higher than 3.20.x

[honghan-wong]

Hi,

We have an issue where if the os is running on a later version with python and pip installed with protobuf version higher than 3.20.x, the auto_backup.py will throw an error like below:

Traceback (most recent call last):
  File "/var/lib/jujubackupall/auto_backup.py", line 42, in <module>
    from jujubackupall.process import BackupProcessor  # noqa E402
  File "/var/lib/juju/agents/unit-juju-backup-all-0/charm/venv/jujubackupall/process.py", line 24, in <module>
    from juju.application import Application
  File "/var/lib/juju/agents/unit-juju-backup-all-0/charm/venv/juju/application.py", line 10, in <module>
    from . import jasyncio, model, tag, utils
  File "/var/lib/juju/agents/unit-juju-backup-all-0/charm/venv/juju/model.py", line 29, in <module>
    from .client import client, connector
  File "/var/lib/juju/agents/unit-juju-backup-all-0/charm/venv/juju/client/connector.py", line 9, in <module>
    import macaroonbakery.httpbakery as httpbakery
  File "/usr/lib/python3/dist-packages/macaroonbakery/httpbakery/__init__.py", line 3, in <module>
    from ._client import (
  File "/usr/lib/python3/dist-packages/macaroonbakery/httpbakery/_client.py", line 7, in <module>
    import macaroonbakery.bakery as bakery
  File "/usr/lib/python3/dist-packages/macaroonbakery/bakery/__init__.py", line 75, in <module>
    from ._oven import (
  File "/usr/lib/python3/dist-packages/macaroonbakery/bakery/_oven.py", line 29, in <module>
    from ._internal import id_pb2
  File "/usr/lib/python3/dist-packages/macaroonbakery/bakery/_internal/id_pb2.py", line 35, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/lib/python3/dist-packages/google/protobuf/descriptor.py", line 553, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

This is due to a change in protobuf release, link.

Downgrading python protobuf lib is a workaround to this. pip install -Iv protobuf==3.20.1

We are not sure the ideal way to handle the python protobuf lib version to not breaking older version. If the new protoc is backward compatible then we should be good to just compile the charm with a later protobuf version. However if it is not, we need to think of a way to handle old protobuf version, like running charm in venv.

[samuelallan72]

Thanks for the report! I'm doing some investigating now.

The dependency chain looks like: charm-juju-backup-all -> jujubackupall == v1.1.0 (direct from github) -> juju -> macaroonbakery -> protobuf >= 3.20.0 .

So I guess the root cause is macaroonbakery not being compatible with protobuf 4.x. I'll start looking into what will be required to fix this upstream. :)

[samuelallan72]

However if it is not, we need to think of a way to handle old protobuf version, like running charm in venv.

Charmcraft should be shipping all the dependencies with the charm, so a quick workaround may be to pin protobuf==3.20.1 in the charm requirements.

... although this is weird:

  File "/var/lib/juju/agents/unit-juju-backup-all-0/charm/venv/juju/client/connector.py", line 9, in <module>
    import macaroonbakery.httpbakery as httpbakery
  File "/usr/lib/python3/dist-packages/macaroonbakery/httpbakery/__init__.py", line 3, in <module>
    from ._client import (

It's using macaroonbakery from the system, even though it's included with the charm. :thinking:

[samuelallan72]

@honghan-wong could you add more details please to help with reproducing this? Charm revision / channel, base, when was this traceback seen?

[honghan-wong]

Hi Samuel, the charm we are using is juju-backup-all latest/edge 14 OS Ubuntu 22.04.4 LTS kernel 5.15.0-112-generic It was first seen when we are doing a handover, happened around Jun 28.

The reproduce step is just deploy the charm on a server, and the charm will goes into the error state.

[samuelallan72]

I discovered that the auto_backup script will favour importing packages from the system libraries, instead of from the charm. This means that if there are incompatible versions installed system-wide, it will use those and crash. I opened https://github.com/canonical/charm-juju-backup-all/pull/46 to fix the issue.

However, I'm not sure about the root cause of this particular issue. It's not clear why python3-macaroonbakery and python3-protobuf are installed systemwide with apt. Also I couldn't reproduce the issue with latest/edge. Note also that protobuf 5.26.1 is shipped with the charm env.

[samuelallan72]

Confirming that #46 fixed the issue. :) Thanks for the report and the help confirming the fix @honghan-wong !