search

juju / charm-tools

Tools for charm authors and maintainers
Other
42 stars 64 forks source link

Built charm and Python dependency issues #647

Closed hloeung closed 1 year ago

hloeung commented 1 year ago

Checklist

What version am I running?

I ran the following command: snap info charm and got the following ouput:

name:      charm
summary:   charm-tools
publisher: Canonical✓
store-url: https://snapcraft.io/charm
contact:   https://discourse.juju.is/c/charming
license:   GPL-3.0
description: |
  Tools for building and maintaining reactive charms.
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     latest/stable
refresh-date: 8 days ago, at 08:45 AEDT
channels:
  3.x/stable:       3.0.4               2022-10-09 (687)  28MB classic
  3.x/candidate:    ↑
  3.x/beta:         ↑
  3.x/edge:         ↑
  latest/stable:    3.0.4               2022-10-09 (687)  28MB classic
  latest/candidate: ↑
  latest/beta:      ↑
  latest/edge:      3.0.4               2022-10-09 (687)  28MB classic
  2.x/stable:       2.8.2               2022-09-28 (609) 119MB classic
  2.x/candidate:    ↑
  2.x/beta:         ↑
  2.x/edge:         2.8.4+git-9-98edcd3 2022-09-28 (664) 119MB classic
installed:          3.0.4                          (687)  28MB classic

I am using: Ubuntu 22.10

Issue/Feature

We need to support various versions of Ubuntu in some of our charms. For instance, the autocert charm (https://charmhub.io/autocert-charmers-autocert) needs to support Jammy all the way back to Xenial.

Unfortunately, building the charms as is results in failures to execute on older series/releases. Pinning via wheelhouse.txt works for some series but still breaks on others. This is a reactive charm and uses layer-basic which now has more complicated pinning, see https://github.com/juju-solutions/layer-basic/commit/fb767dcf0786d1d5364199bb3b40bdc86518b45b

I think charm-tools should bundle multiple versions of Python packages and let wheelhouse select the appropriate version to install. For example, layer-basic's wheelhouse.txt has the following:

MarkupSafe<2.0.0;python_version < '3.6'
MarkupSafe<2.1.0;python_version == '3.6' # Just for python 3.6
MarkupSafe;python_version >= '3.7' # newer pythons

So a charmcraft pack should bundle MarkupSafe 1.1.1, 2.0.1, and the latest, 2.1.1.

I expect/expected the following

juju upgrade-charm and all units should successfully execute their hooks.

What I got

juju upgrade-charm on various deployed autocert applications (because https://bugs.launchpad.net/juju/+bug/1992881), one for each series and we get all kinds of Python dependency issues resulting in failure to run charm hooks.

fnordahl commented 1 year ago

Hello, Haw,

Python dependency management across a wide set of Python versions and architectures is indeed a non-trivial activity.

The preferred method of building reactive charms is to use charmcraft and its bases support together with charm-tools.

This allows you to build multiple artifacts for the same charm and also define which one is suited to run on which versions of the target operating system. charmhub.io has gained the same capabilities and will allow Juju to select the correct charm. Launchpad charm recipes is also ready to help you build across the defined matrix of series and architectures.

When building with charmcraft dependency managements can be further enhanced by using the support for binary builds (#620), which allows pip's dependency resolution engine to work for python build dependencies.

As for your specific request:

I think charm-tools should bundle multiple versions of Python packages and let wheelhouse select the appropriate version to install. For example, layer-basic's wheelhouse.txt has the following:

To achieve this a single build instance would have to have multiple versions of python and its dependencies installed and would complicate a already complicated process. As laid out above, the path chosen is to build multiple artifacts using build/run bases and the infrastructure to support it in Launchpad and Charmhub.

ajkavanagh commented 1 year ago

So a charmcraft pack should bundle MarkupSafe 1.1.1, 2.0.1, and the latest, 2.1.1.

This would actually require layer-basic to work out which one to install, as it installs everything in the wheelhouse/ directory. Or there would need to be multiple wheelhouse/ directories, one for each python version.

It's actually just easier to build separate charms for each base (16.04, 18.04, 20.04, 22.04) and let the charmhub serve up the 'right' charm when juju requests a charm for a particular series. i.e. the latest/stable channel can have multiple revisions 'active' if they have different bases, which allows older bases to co-exist with newer ones. As @fnordahl mentioned, this is possible with charmcraft and the build-on, run-on sections in the bases key. e.g. see https://opendev.org/openstack/charm-octavia/src/commit/a70543515682ccae99f64a46653bc9d6601083e5/charmcraft.yaml#L35 as an example.

As you noticed with layer-basic, it is possible to have a wheelhouse.txt that declares different module versions for different python versions, and then using bases to build the charm on different Ubuntu versions for the different versions. e.g. Xenial, is py35, bionic is py36, focal is py38, and jammy is py310.

In wheelhouse.txt:

some_module < 1; python_version == '3.5'  # xenial
some_module < 2; python_version == '3.6'  # bionic
some_module < 3; python_version == '3.8'  # focal
some_module < 4; python_version >= '3.10' # jammy and later

If using a build.lock file, leave those modules out of the build.lock.

However, I would recommend that you split your repository into separate branches and have different build recipes for xenial, etc. The use the launchpad recipes to build multiple charms with charmcraft, one for each series, using the reactive plugin and push to the relevant channel(s). As the charms would have different bases they could all push to latest/stable.

Trying to get a single source that can build against all the different versions of python (particularly as python 3.5 and 3.6 are now deprecated, which means that modules are dropping support for them) will probably be more of a maintenance burden, particularly when the older versions of the charms are less likely to require changes.