"charm build" fails when wheelhouse.txt refers to cryptography<3.3,>=3.2 due to cffi dependency

Vultaire commented 3 years ago

Hello,

I've encountered an issue with charm build when building with the Python cryptography library under certain circumstances.

I've created a demo repository which provides exact code which reproduces the issue: https://github.com/Vultaire/charm-cffi-bug-sscce

Basically, it appears that the include paths used when installing a dependency of cryptography, cffi, are not quite adequate to match the header files provided by the charm snap. This results in the build failing since gcc cannot find the appropriate header file.

Please see the above mentioned demo repository for further details. Command executed: "charm build --debug charm-test"

Checklist

[X] Confirmed this is an issue with charm-tools, not charmstore-client
[X] Provide versions of tools used
[X] Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: snap info charm and got the following ouput:

name:      charm
summary:   charm and charm-tools
publisher: Canonical✓
store-url: https://snapcraft.io/charm
contact:   https://discourse.juju.is/c/charming
license:   unset
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     latest/stable
refresh-date: 2021-03-29
channels:
  latest/stable:    2.8.2               2021-02-01 (609) 119MB classic
  latest/candidate: 2.8.2               2021-02-01 (609) 119MB classic
  latest/beta:      2.8.2               2021-02-01 (609) 119MB classic
  latest/edge:      2.8.3+git-1-736b1ad 2021-02-23 (620) 119MB classic
installed:          2.8.2                          (609) 119MB classic

I am using: Ubuntu Focal

Issue/Feature

I expect/expected the following

I expected the charm to build without errors.

What I got

The charm failed to build since gcc could not find a required header file. The snap does provide this header file, but an appropriate -I parameter to gcc was not provided, thus gcc could not find it.

Quick excerpt:

      creating build/temp.linux-x86_64-3.6/c
      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/tmp/tmpqba_1gu6/include -I/snap/charm/609/usr/include/python3.6m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.6/c/_cffi_backend.o
      In file included from /snap/charm/609/usr/include/python3.6m/Python.h:8,
                       from c/_cffi_backend.c:2:
      /snap/charm/609/usr/include/python3.6m/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: そのようなファイルやディレクトリはありません
          3 | #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

For full details, please see the demo repository mentioned near the top of this message.

johnsca commented 3 years ago

This is a long-standing issue that stems from the old charm store not having the ability to do per-arch charms, meaning dependencies which include compiled code like this need to be handled differently. If possible, it's strongly recommended to move to the Operator framework, the charmcraft tool, and CharmHub.io which has native support for architecture-aware charm builds.

If for some reason you can't yet migrate your charm, you can use the python_packages layer config to arrange for the package to be built and installed at install time, however this means that your charm will need to be able to talk to PyPI when deployed. Another option would be to attach the arch-specific library as a binary wheel resource for each supported arch, but there's no framework-provided helpers for doing that (although it's not terribly complicated; just have an install / upgrade hook handler that calls resource-get and pip install on the appropriate resource for the arch it's running on).

Vultaire commented 3 years ago

Thanks John - python_packages is perfect here I think; we already are using it for one Python dependency.

I think we can close this.

juju / charm-tools