charm build results in c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory

xavpaice commented 5 years ago

What version am I running?

I ran the following command: snap info charm and got the following output:

name:      charm
summary:   charm and charm-tools
publisher: Cory Johns (johnsca)
contact:   juju@lists.ubuntu.com
license:   unset
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     stable
refresh-date: yesterday at 14:53 NZDT
channels:                                    
  stable:    2.4.5+git-1-gd62c072 (298) 47MB classic
  candidate: 2.4.5+git-1-gd62c072 (298) 47MB classic
  beta:      2.4.5+git-1-gd62c072 (298) 47MB classic
  edge:      2.4.5+git-1-gd62c072 (298) 47MB classic
installed:   2.4.5+git-1-gd62c072 (298) 47MB classic

I am using: Ubuntu Bionic

Issue

When running charm build on my charm, the dependencies fail to compile and build fails as a result.

This shows up when I have a charm that has the following in wheelhouse.txt:

cffi>=1.7,!=1.11.3
openstacksdk

I expect/expected the following

charm build --no-local-layers -v should complete without error. If I remove cffi and openstacksdk from wheelhouse.txt it's all fine, even if I put different packages there.

What I got

xav@xplt:~/charms/charm-openstack-service-checks$ charm build --no-local-layers -v
build: Destination charm directory: /home/xav/charms/builds/openstack-service-checks
build: {
  "INTERFACE_PATH": "/home/xav/charms/interfaces",
  "JUJU_REPOSITORY": "/home/xav/charms",
  "LAYER_PATH": "/home/xav/charms/layers",
  "_charm": ".",
  "_charm_metadata": {
    "description": "OpenStack Services NRPE Checks\n",
    "extra-bindings": {
      "admin": null,
      "internal": null,
      "public": null
    },
    "maintainer": "Jill Rouleau <jill.rouleau@canonical.com>",
    "name": "openstack-service-checks",
    "provides": {
      "nrpe-external-master": {
        "interface": "nrpe-external-master",
        "optional": true,
        "scope": "container"
      }
    },
    "requires": {
      "identity-credentials": {
        "interface": "keystone-credentials",
        "optional": true
      }
    },
    "series": [
      "xenial",
      "trusty",
      "bionic"
    ],
    "subordinate": false,
    "summary": "OpenStack Services NRPE Checks",
    "tags": [
      "openstack",
      "ops",
      "monitoring"
    ]
  },
  "_name": "openstack-service-checks",
  "_top_layer": null,
  "_warned_home": false,
  "config": "BuildConfig({'_tactics': [], 'configured': False})",
  "deps": "/home/xav/charms/deps",
  "description": false,
  "force": false,
  "hide_metrics": false,
  "interface_service": null,
  "layer_index": "https://juju.github.io/layer-index/",
  "log_level": 10,
  "no_local_layers": true,
  "output_dir": "/home/xav/charms",
  "repo": "/home/xav/charms/builds",
  "report": false,
  "series": null,
  "target_dir": "/home/xav/charms/builds/openstack-service-checks",
  "verbose": true,
  "wheelhouse_overrides": null
}
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/basic.json
charmtools.build.fetchers: Found repo: https://github.com/juju-solutions/layer-basic.git
charmtools.fetchers: git clone https://github.com/juju-solutions/layer-basic.git /home/xav/charms/deps/layer/tmp434fio6o: b"Cloning into '/home/xav/charms/deps/layer/tmp434fio6o'...\nwarning: templates not found /snap/charm/298/share/git-core/templates\n"
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/options.json
charmtools.build.fetchers: Found repo: https://github.com/juju-solutions/layer-options.git
charmtools.fetchers: git clone https://github.com/juju-solutions/layer-options.git /home/xav/charms/deps/layer/tmphisz3j5j: b"Cloning into '/home/xav/charms/deps/layer/tmphisz3j5j'...\nwarning: templates not found /snap/charm/298/share/git-core/templates\n"
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/interfaces/nrpe-external-master.json
charmtools.build.fetchers: Found repo: https://github.com/cmars/nrpe-external-master-interface
charmtools.fetchers: git clone https://github.com/cmars/nrpe-external-master-interface /home/xav/charms/deps/interface/tmptoti1ta4: b"Cloning into '/home/xav/charms/deps/interface/tmptoti1ta4'...\nwarning: templates not found /snap/charm/298/share/git-core/templates\n"
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/interfaces/keystone-credentials.json
charmtools.build.fetchers: Found repo: https://github.com/openstack/charm-interface-keystone-credentials
charmtools.fetchers: git clone https://github.com/openstack/charm-interface-keystone-credentials /home/xav/charms/deps/interface/tmpv3u1bwfs: b"Cloning into '/home/xav/charms/deps/interface/tmpv3u1bwfs'...\nwarning: templates not found /snap/charm/298/share/git-core/templates\n"
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/apt.json
charmtools.build.fetchers: Found repo: https://git.launchpad.net/layer-apt
charmtools.fetchers: git clone https://git.launchpad.net/layer-apt /home/xav/charms/deps/layer/tmpmqozn2du: b"Cloning into '/home/xav/charms/deps/layer/tmpmqozn2du'...\nwarning: templates not found /snap/charm/298/share/git-core/templates\n"
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/layers/status.json
charmtools.build.fetchers: Found repo: https://github.com/juju-solutions/layer-status
charmtools.fetchers: git clone https://github.com/juju-solutions/layer-status /home/xav/charms/deps/layer/tmpn9ggiep2: b"Cloning into '/home/xav/charms/deps/layer/tmpn9ggiep2'...\nwarning: templates not found /snap/charm/298/share/git-core/templates\n"
build: Processing layer: layer:options
build: Processing layer: layer:basic
build: Processing layer: layer:status
build: Processing layer: layer:apt
build: Processing layer: openstack-service-checks (from .)
build: Processing interface: nrpe-external-master
build: Processing interface: keystone-credentials
charmtools.build.tactics: Copying openstack-service-checks: /home/xav/charms/builds/openstack-service-checks/README.md
charmtools.build.tactics: Creating /home/xav/charms/builds/openstack-service-checks/lib
charmtools.build.tactics: Creating /home/xav/charms/builds/openstack-service-checks/lib/charms
charmtools.build.tactics: Creating /home/xav/charms/builds/openstack-service-checks/lib/charms/layer
charmtools.build.tactics: Copying options: /home/xav/charms/builds/openstack-service-checks/lib/charms/layer/options.py
charmtools.build.tactics: Copying openstack-service-checks: /home/xav/charms/builds/openstack-service-checks/LICENSE
charmtools.build.tactics: Creating /home/xav/charms/builds/openstack-service-checks/bin
charmtools.build.tactics: Copying options: /home/xav/charms/builds/openstack-service-checks/bin/layer_option
charmtools.build.tactics: Copying openstack-service-checks: /home/xav/charms/builds/openstack-service-checks/.gitignore
charmtools.build.tactics: Creating /home/xav/charms/builds/openstack-service-checks/hooks
charmtools.build.tactics: Copying basic: /home/xav/charms/builds/openstack-service-checks/hooks/hook.template
utils: b"Already using interpreter /snap/charm/298/usr/bin/python3\nUsing base prefix '/snap/charm/298/usr'\nNew python executable in /tmp/tmpv27o2kzh/bin/python3\nAlso creating executable in /tmp/tmpv27o2kzh/bin/python\nInstalling setuptools, pip, wheel...\ndone.\n"
utils: process: virtualenv --python python3 /tmp/tmpv27o2kzh (0)
utils: b"Already using interpreter /snap/charm/298/usr/bin/python3\nUsing base prefix '/snap/charm/298/usr'\nNew python executable in /tmp/tmp9fm9wh_z/bin/python3\nAlso creating executable in /tmp/tmp9fm9wh_z/bin/python\nInstalling setuptools, pip, wheel...\ndone.\n"
utils: process: virtualenv --python python3 /tmp/tmp9fm9wh_z (0)
utils: b'Collecting pip<8.2.0,>=7.0.0 (from -r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/e7/a8/7556133689add8d1a54c0b14aeff0acb03c64707ce100ecd53934da1aa13/pip-8.1.2.tar.gz\n  Saved /tmp/tmp_wjw9att/pip-8.1.2.tar.gz\nCollecting setuptools<=39.0.1,>24.3 (from -r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/72/c2/c09362ab29338413ab687b47dab03bab4a792e2bbb727a1eb5e0a88e3b86/setuptools-39.0.1.zip\n  Saved /tmp/tmp_wjw9att/setuptools-39.0.1.zip\nCollecting setuptools-scm<=1.17.0 (from -r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 3))\n  Using cached https://files.pythonhosted.org/packages/a7/52/5f84da9ee2780682795550ddea20bec3e604a8a82600f4e5d3a6ca0bcbcd/setuptools_scm-1.17.0.tar.gz\n  Saved /tmp/tmp_wjw9att/setuptools_scm-1.17.0.tar.gz\nCollecting charmhelpers<1.0.0,>=0.4.0 (from -r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/fc/4c/9d49c153d24ff93ecc9fce8bb5d4617d43b6e00922a77c2b35e91851fbf5/charmhelpers-0.19.5.tar.gz\n  Saved /tmp/tmp_wjw9att/charmhelpers-0.19.5.tar.gz\nCollecting charms.reactive<2.0.0,>=0.1.0 (from -r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 5))\n  Using cached https://files.pythonhosted.org/packages/b4/54/a92cd94abcbd206edc7cc702b9a923bc7e9f001be7d1a391bf09ecf44524/charms.reactive-1.1.2.tar.gz\n  Saved /tmp/tmp_wjw9att/charms.reactive-1.1.2.tar.gz\nCollecting netaddr (from charmhelpers<1.0.0,>=0.4.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/0c/13/7cbb180b52201c07c796243eeff4c256b053656da5cfe3916c3f5b57b3a0/netaddr-0.7.19.tar.gz\n  Saved /tmp/tmp_wjw9att/netaddr-0.7.19.tar.gz\nCollecting PyYAML (from charmhelpers<1.0.0,>=0.4.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz\n  Saved /tmp/tmp_wjw9att/PyYAML-3.13.tar.gz\nCollecting Tempita (from charmhelpers<1.0.0,>=0.4.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/56/c8/8ed6eee83dbddf7b0fc64dd5d4454bc05e6ccaafff47991f73f2894d9ff4/Tempita-0.5.2.tar.gz\n  Saved /tmp/tmp_wjw9att/Tempita-0.5.2.tar.gz\nCollecting Jinja2 (from charmhelpers<1.0.0,>=0.4.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/56/e6/332789f295cf22308386cf5bbd1f4e00ed11484299c5d7383378cf48ba47/Jinja2-2.10.tar.gz\n  Saved /tmp/tmp_wjw9att/Jinja2-2.10.tar.gz\nCollecting six (from charmhelpers<1.0.0,>=0.4.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz\n  Saved /tmp/tmp_wjw9att/six-1.12.0.tar.gz\nCollecting pyaml (from charms.reactive<2.0.0,>=0.1.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 5))\n  Using cached https://files.pythonhosted.org/packages/e3/a2/e430a32409d059c56d2c5c09af35985250b83b69da90eb358bf4ad6b974c/pyaml-18.11.0.tar.gz\n  Saved /tmp/tmp_wjw9att/pyaml-18.11.0.tar.gz\nCollecting MarkupSafe>=0.23 (from Jinja2->charmhelpers<1.0.0,>=0.4.0->-r /home/xav/charms/deps/layer/basic/wheelhouse.txt (line 4))\n  Using cached https://files.pythonhosted.org/packages/ac/7e/1b4c2e05809a4414ebce0892fe1e32c14ace86ca7d50c70f00979ca9b3a3/MarkupSafe-1.1.0.tar.gz\n  Saved /tmp/tmp_wjw9att/MarkupSafe-1.1.0.tar.gz\nSuccessfully downloaded pip setuptools setuptools-scm charmhelpers charms.reactive netaddr PyYAML Tempita Jinja2 six pyaml MarkupSafe\n'
utils: process: bash -c . /tmp/tmp9fm9wh_z/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmp_wjw9att -r /home/xav/charms/deps/layer/basic/wheelhouse.txt (0)
utils: b'Collecting cffi!=1.11.3,>=1.7 (from -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz\n  Saved /tmp/tmpwax5xvuy/cffi-1.11.5.tar.gz\nCollecting openstacksdk (from -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/df/66/26a943b8073db82ff368e8712fe76107f2ab6f24c432a5a9fe5893511fdd/openstacksdk-0.22.0.tar.gz\n  Saved /tmp/tmpwax5xvuy/openstacksdk-0.22.0.tar.gz\nCollecting pycparser (from cffi!=1.11.3,>=1.7->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 1))\n  Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz\n  Saved /tmp/tmpwax5xvuy/pycparser-2.19.tar.gz\nCollecting pbr!=2.1.0,>=2.0.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/33/07/6e68a96ff240a0e7bb1f6e21093532386a98a82d56512e1e3da6d125f7aa/pbr-5.1.1.tar.gz\n  Saved /tmp/tmpwax5xvuy/pbr-5.1.1.tar.gz\nCollecting PyYAML>=3.12 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz\n  Saved /tmp/tmpwax5xvuy/PyYAML-3.13.tar.gz\nCollecting appdirs>=1.3.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/48/69/d87c60746b393309ca30761f8e2b49473d43450b150cb08f3c6df5c11be5/appdirs-1.4.3.tar.gz\n  Saved /tmp/tmpwax5xvuy/appdirs-1.4.3.tar.gz\nCollecting requestsexceptions>=1.2.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/82/ed/61b9652d3256503c99b0b8f145d9c8aa24c514caff6efc229989505937c1/requestsexceptions-1.4.0.tar.gz\n  Saved /tmp/tmpwax5xvuy/requestsexceptions-1.4.0.tar.gz\nCollecting jsonpatch!=1.20,>=1.16 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/9a/7d/bcf203d81939420e1aaf7478a3efce1efb8ccb4d047a33cb85d7f96d775e/jsonpatch-1.23.tar.gz\n  Saved /tmp/tmpwax5xvuy/jsonpatch-1.23.tar.gz\nCollecting six>=1.10.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz\n  Saved /tmp/tmpwax5xvuy/six-1.12.0.tar.gz\nCollecting os-service-types>=1.2.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/ad/1b/7bc2b1f91e5e4c51c40fcbe6d4ebe068f8a96490a0e12f480f98940f4ba4/os-service-types-1.4.0.tar.gz\n  Saved /tmp/tmpwax5xvuy/os-service-types-1.4.0.tar.gz\nCollecting keystoneauth1>=3.11.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/f8/db/4ef4ce85a3136d193303f519d42af24f230b13d4e42ea449ceb274f8cd08/keystoneauth1-3.11.2.tar.gz\n  Saved /tmp/tmpwax5xvuy/keystoneauth1-3.11.2.tar.gz\nCollecting munch>=2.1.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/68/f4/260ec98ea840757a0da09e0ed8135333d59b8dfebe9752a365b04857660a/munch-2.3.2.tar.gz\n  Saved /tmp/tmpwax5xvuy/munch-2.3.2.tar.gz\nCollecting decorator>=3.4.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/6f/24/15a229626c775aae5806312f6bf1e2a73785be3402c0acdec5dbddd8c11e/decorator-4.3.0.tar.gz\n  Saved /tmp/tmpwax5xvuy/decorator-4.3.0.tar.gz\nCollecting jmespath>=0.9.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/e5/21/795b7549397735e911b032f255cff5fb0de58f96da794274660bca4f58ef/jmespath-0.9.3.tar.gz\n  Saved /tmp/tmpwax5xvuy/jmespath-0.9.3.tar.gz\nCollecting iso8601>=0.1.11 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/45/13/3db24895497345fb44c4248c08b16da34a9eb02643cea2754b21b5ed08b0/iso8601-0.1.12.tar.gz\n  Saved /tmp/tmpwax5xvuy/iso8601-0.1.12.tar.gz\nCollecting netifaces>=0.10.4 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/81/39/4e9a026265ba944ddf1fea176dbb29e0fe50c43717ba4fcf3646d099fe38/netifaces-0.10.7.tar.gz\n  Saved /tmp/tmpwax5xvuy/netifaces-0.10.7.tar.gz\nCollecting dogpile.cache<0.7.0,>=0.6.2 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/73/bf/0cbed594e4f0f9360bfb98e7276131bf32e1af1d15e6c11a9dd8bd93a12f/dogpile.cache-0.6.8.tar.gz\n  Saved /tmp/tmpwax5xvuy/dogpile.cache-0.6.8.tar.gz\nCollecting cryptography>=2.1 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))\n  Using cached https://files.pythonhosted.org/packages/f3/39/d3904df7c56f8654691c4ae1bdb270c1c9220d6da79bd3b1fbad91afd0e1/cryptography-2.4.2.tar.gz\n  Saved /tmp/tmpwax5xvuy/cryptography-2.4.2.tar.gz\n  Installing build dependencies: started\n  Installing build dependencies: finished with status \'error\'\n  Complete output from command /tmp/tmpv27o2kzh/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k9z2vsvm --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools>=18.5 wheel "cffi>=1.7,!=1.11.3; python_implementation != \'PyPy\'":\n  Collecting setuptools>=18.5\n    Using cached https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b/setuptools-40.6.3.zip\n  Collecting wheel\n    Using cached https://files.pythonhosted.org/packages/d8/55/221a530d66bf78e72996453d1e2dedef526063546e131d70bed548d80588/wheel-0.32.3.tar.gz\n  Collecting cffi!=1.11.3,>=1.7\n    Using cached https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz\n  Collecting pycparser (from cffi!=1.11.3,>=1.7)\n    Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz\n  Skipping bdist_wheel for setuptools, due to binaries being disabled for it.\n  Skipping bdist_wheel for wheel, due to binaries being disabled for it.\n  Skipping bdist_wheel for cffi, due to binaries being disabled for it.\n  Skipping bdist_wheel for pycparser, due to binaries being disabled for it.\n  Installing collected packages: setuptools, wheel, pycparser, cffi\n    Running setup.py install for setuptools: started\n      Running setup.py install for setuptools: finished with status \'done\'\n    Running setup.py install for wheel: started\n      Running setup.py install for wheel: finished with status \'done\'\n    Running setup.py install for pycparser: started\n      Running setup.py install for pycparser: finished with status \'done\'\n    Running setup.py install for cffi: started\n      Running setup.py install for cffi: finished with status \'error\'\n      Complete output from command /tmp/tmpv27o2kzh/bin/python3 -u -c "import setuptools, tokenize;__file__=\'/tmp/pip-install-9blhp77j/cffi/setup.py\';f=getattr(tokenize, \'open\', open)(__file__);code=f.read().replace(\'\\r\\n\', \'\\n\');f.close();exec(compile(code, __file__, \'exec\'))" install --record /tmp/pip-record-22e5dy1o/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-k9z2vsvm --compile --install-headers /tmp/tmpv27o2kzh/include/site/python3.5/cffi:\n      running install\n      running build\n      running build_py\n      creating build\n      creating build/lib.linux-x86_64-3.5\n      creating build/lib.linux-x86_64-3.5/cffi\n      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/verifier.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/lock.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/cparser.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/__init__.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/model.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/api.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/error.py -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.5/cffi\n      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.5/cffi\n      running build_ext\n      building \'_cffi_backend\' extension\n      creating build/temp.linux-x86_64-3.5\n      creating build/temp.linux-x86_64-3.5/c\n      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/snap/charm/298/usr/include/python3.5m -I/tmp/tmpv27o2kzh/include/python3.5m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.5/c/_cffi_backend.o\n      c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory\n       #include <Python.h>\n                ^~~~~~~~~~\n      compilation terminated.\n      error: command \'x86_64-linux-gnu-gcc\' failed with exit status 1\n  \n      ----------------------------------------\n  Command "/tmp/tmpv27o2kzh/bin/python3 -u -c "import setuptools, tokenize;__file__=\'/tmp/pip-install-9blhp77j/cffi/setup.py\';f=getattr(tokenize, \'open\', open)(__file__);code=f.read().replace(\'\\r\\n\', \'\\n\');f.close();exec(compile(code, __file__, \'exec\'))" install --record /tmp/pip-record-22e5dy1o/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-k9z2vsvm --compile --install-headers /tmp/tmpv27o2kzh/include/site/python3.5/cffi" failed with error code 1 in /tmp/pip-install-9blhp77j/cffi/\n  \n  ----------------------------------------\nCommand "/tmp/tmpv27o2kzh/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k9z2vsvm --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools>=18.5 wheel "cffi>=1.7,!=1.11.3; python_implementation != \'PyPy\'"" failed with error code 1 in None\n'
utils: process: bash -c . /tmp/tmpv27o2kzh/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmpwax5xvuy -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (1)
Collecting cffi!=1.11.3,>=1.7 (from -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz
  Saved /tmp/tmpwax5xvuy/cffi-1.11.5.tar.gz
Collecting openstacksdk (from -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/df/66/26a943b8073db82ff368e8712fe76107f2ab6f24c432a5a9fe5893511fdd/openstacksdk-0.22.0.tar.gz
  Saved /tmp/tmpwax5xvuy/openstacksdk-0.22.0.tar.gz
Collecting pycparser (from cffi!=1.11.3,>=1.7->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
  Saved /tmp/tmpwax5xvuy/pycparser-2.19.tar.gz
Collecting pbr!=2.1.0,>=2.0.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/33/07/6e68a96ff240a0e7bb1f6e21093532386a98a82d56512e1e3da6d125f7aa/pbr-5.1.1.tar.gz
  Saved /tmp/tmpwax5xvuy/pbr-5.1.1.tar.gz
Collecting PyYAML>=3.12 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz
  Saved /tmp/tmpwax5xvuy/PyYAML-3.13.tar.gz
Collecting appdirs>=1.3.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/48/69/d87c60746b393309ca30761f8e2b49473d43450b150cb08f3c6df5c11be5/appdirs-1.4.3.tar.gz
  Saved /tmp/tmpwax5xvuy/appdirs-1.4.3.tar.gz
Collecting requestsexceptions>=1.2.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/82/ed/61b9652d3256503c99b0b8f145d9c8aa24c514caff6efc229989505937c1/requestsexceptions-1.4.0.tar.gz
  Saved /tmp/tmpwax5xvuy/requestsexceptions-1.4.0.tar.gz
Collecting jsonpatch!=1.20,>=1.16 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/9a/7d/bcf203d81939420e1aaf7478a3efce1efb8ccb4d047a33cb85d7f96d775e/jsonpatch-1.23.tar.gz
  Saved /tmp/tmpwax5xvuy/jsonpatch-1.23.tar.gz
Collecting six>=1.10.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz
  Saved /tmp/tmpwax5xvuy/six-1.12.0.tar.gz
Collecting os-service-types>=1.2.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/ad/1b/7bc2b1f91e5e4c51c40fcbe6d4ebe068f8a96490a0e12f480f98940f4ba4/os-service-types-1.4.0.tar.gz
  Saved /tmp/tmpwax5xvuy/os-service-types-1.4.0.tar.gz
Collecting keystoneauth1>=3.11.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/f8/db/4ef4ce85a3136d193303f519d42af24f230b13d4e42ea449ceb274f8cd08/keystoneauth1-3.11.2.tar.gz
  Saved /tmp/tmpwax5xvuy/keystoneauth1-3.11.2.tar.gz
Collecting munch>=2.1.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/68/f4/260ec98ea840757a0da09e0ed8135333d59b8dfebe9752a365b04857660a/munch-2.3.2.tar.gz
  Saved /tmp/tmpwax5xvuy/munch-2.3.2.tar.gz
Collecting decorator>=3.4.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/6f/24/15a229626c775aae5806312f6bf1e2a73785be3402c0acdec5dbddd8c11e/decorator-4.3.0.tar.gz
  Saved /tmp/tmpwax5xvuy/decorator-4.3.0.tar.gz
Collecting jmespath>=0.9.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/e5/21/795b7549397735e911b032f255cff5fb0de58f96da794274660bca4f58ef/jmespath-0.9.3.tar.gz
  Saved /tmp/tmpwax5xvuy/jmespath-0.9.3.tar.gz
Collecting iso8601>=0.1.11 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/45/13/3db24895497345fb44c4248c08b16da34a9eb02643cea2754b21b5ed08b0/iso8601-0.1.12.tar.gz
  Saved /tmp/tmpwax5xvuy/iso8601-0.1.12.tar.gz
Collecting netifaces>=0.10.4 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/81/39/4e9a026265ba944ddf1fea176dbb29e0fe50c43717ba4fcf3646d099fe38/netifaces-0.10.7.tar.gz
  Saved /tmp/tmpwax5xvuy/netifaces-0.10.7.tar.gz
Collecting dogpile.cache<0.7.0,>=0.6.2 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/73/bf/0cbed594e4f0f9360bfb98e7276131bf32e1af1d15e6c11a9dd8bd93a12f/dogpile.cache-0.6.8.tar.gz
  Saved /tmp/tmpwax5xvuy/dogpile.cache-0.6.8.tar.gz
Collecting cryptography>=2.1 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/f3/39/d3904df7c56f8654691c4ae1bdb270c1c9220d6da79bd3b1fbad91afd0e1/cryptography-2.4.2.tar.gz
  Saved /tmp/tmpwax5xvuy/cryptography-2.4.2.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /tmp/tmpv27o2kzh/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k9z2vsvm --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools>=18.5 wheel "cffi>=1.7,!=1.11.3; python_implementation != 'PyPy'":
  Collecting setuptools>=18.5
    Using cached https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b/setuptools-40.6.3.zip
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/d8/55/221a530d66bf78e72996453d1e2dedef526063546e131d70bed548d80588/wheel-0.32.3.tar.gz
  Collecting cffi!=1.11.3,>=1.7
    Using cached https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz
  Collecting pycparser (from cffi!=1.11.3,>=1.7)
    Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
  Skipping bdist_wheel for setuptools, due to binaries being disabled for it.
  Skipping bdist_wheel for wheel, due to binaries being disabled for it.
  Skipping bdist_wheel for cffi, due to binaries being disabled for it.
  Skipping bdist_wheel for pycparser, due to binaries being disabled for it.
  Installing collected packages: setuptools, wheel, pycparser, cffi
    Running setup.py install for setuptools: started
      Running setup.py install for setuptools: finished with status 'done'
    Running setup.py install for wheel: started
      Running setup.py install for wheel: finished with status 'done'
    Running setup.py install for pycparser: started
      Running setup.py install for pycparser: finished with status 'done'
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /tmp/tmpv27o2kzh/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-9blhp77j/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-22e5dy1o/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-k9z2vsvm --compile --install-headers /tmp/tmpv27o2kzh/include/site/python3.5/cffi:
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.5
      creating build/lib.linux-x86_64-3.5/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.5/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.5/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-3.5
      creating build/temp.linux-x86_64-3.5/c
      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/snap/charm/298/usr/include/python3.5m -I/tmp/tmpv27o2kzh/include/python3.5m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.5/c/_cffi_backend.o
      c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory
       #include <Python.h>
                ^~~~~~~~~~
      compilation terminated.
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

      ----------------------------------------
  Command "/tmp/tmpv27o2kzh/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-9blhp77j/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-22e5dy1o/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-k9z2vsvm --compile --install-headers /tmp/tmpv27o2kzh/include/site/python3.5/cffi" failed with error code 1 in /tmp/pip-install-9blhp77j/cffi/

  ----------------------------------------
Command "/tmp/tmpv27o2kzh/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k9z2vsvm --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools>=18.5 wheel "cffi>=1.7,!=1.11.3; python_implementation != 'PyPy'"" failed with error code 1 in None

Command failed: bash -c . /tmp/tmpv27o2kzh/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmpwax5xvuy -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt

Out of interest, I tried running the failed command in a virtual env on my box:

xav@xplt:~$ virtualenv --python python3 py36
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/xav/py36/bin/python3
Also creating executable in /home/xav/py36/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
xav@xplt:~$ source py36/bin/activate
(py36) xav@xplt:~$ pip3 download --no-binary :all: -d /tmp/tmpwax5xvuy -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt
Collecting cffi!=1.11.3,>=1.7 (from -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 1))
  File was already downloaded /tmp/tmpwax5xvuy/cffi-1.11.5.tar.gz
Collecting openstacksdk (from -r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/openstacksdk-0.22.0.tar.gz
Collecting pycparser (from cffi!=1.11.3,>=1.7->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 1))
  File was already downloaded /tmp/tmpwax5xvuy/pycparser-2.19.tar.gz
Collecting pbr!=2.1.0,>=2.0.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/pbr-5.1.1.tar.gz
Collecting PyYAML>=3.12 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/PyYAML-3.13.tar.gz
Collecting appdirs>=1.3.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/appdirs-1.4.3.tar.gz
Collecting requestsexceptions>=1.2.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/requestsexceptions-1.4.0.tar.gz
Collecting jsonpatch!=1.20,>=1.16 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/jsonpatch-1.23.tar.gz
Collecting six>=1.10.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/six-1.12.0.tar.gz
Collecting os-service-types>=1.2.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/os-service-types-1.4.0.tar.gz
Collecting keystoneauth1>=3.11.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/keystoneauth1-3.11.2.tar.gz
Collecting munch>=2.1.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/munch-2.3.2.tar.gz
Collecting decorator>=3.4.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/decorator-4.3.0.tar.gz
Collecting jmespath>=0.9.0 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/jmespath-0.9.3.tar.gz
Collecting iso8601>=0.1.11 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/iso8601-0.1.12.tar.gz
Collecting netifaces>=0.10.4 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/netifaces-0.10.7.tar.gz
Collecting dogpile.cache<0.7.0,>=0.6.2 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/dogpile.cache-0.6.8.tar.gz
Collecting cryptography>=2.1 (from openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  File was already downloaded /tmp/tmpwax5xvuy/cryptography-2.4.2.tar.gz
  Installing build dependencies ... done
Collecting jsonpointer>=1.9 (from jsonpatch!=1.20,>=1.16->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/52/e7/246d9ef2366d430f0ce7bdc494ea2df8b49d7a2a41ba51f5655f68cfe85f/jsonpointer-2.0.tar.gz
  Saved /tmp/tmpwax5xvuy/jsonpointer-2.0.tar.gz
Collecting requests>=2.14.2 (from keystoneauth1>=3.11.0->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz
  Saved /tmp/tmpwax5xvuy/requests-2.21.0.tar.gz
Collecting stevedore>=1.20.0 (from keystoneauth1>=3.11.0->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/fd/e3/5bc7226601d6405b588efc9c9f22e3cb0ada577eb957cdcd0ae9093ab6d8/stevedore-1.30.0.tar.gz
  Saved /tmp/tmpwax5xvuy/stevedore-1.30.0.tar.gz
Collecting idna>=2.1 (from cryptography>=2.1->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz
  Saved /tmp/tmpwax5xvuy/idna-2.8.tar.gz
Collecting asn1crypto>=0.21.0 (from cryptography>=2.1->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/fc/f1/8db7daa71f414ddabfa056c4ef792e1461ff655c2ae2928a2b675bfed6b4/asn1crypto-0.24.0.tar.gz
  Saved /tmp/tmpwax5xvuy/asn1crypto-0.24.0.tar.gz
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.14.2->keystoneauth1>=3.11.0->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz
  Saved /tmp/tmpwax5xvuy/chardet-3.0.4.tar.gz
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.14.2->keystoneauth1>=3.11.0->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz
  Saved /tmp/tmpwax5xvuy/urllib3-1.24.1.tar.gz
Collecting certifi>=2017.4.17 (from requests>=2.14.2->keystoneauth1>=3.11.0->openstacksdk->-r /home/xav/charms/charm-openstack-service-checks/wheelhouse.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/55/54/3ce77783acba5979ce16674fc98b1920d00b01d337cfaaf5db22543505ed/certifi-2018.11.29.tar.gz
  Saved /tmp/tmpwax5xvuy/certifi-2018.11.29.tar.gz
Successfully downloaded cffi openstacksdk pycparser pbr PyYAML appdirs requestsexceptions jsonpatch six os-service-types keystoneauth1 munch decorator jmespath iso8601 netifaces dogpile.cache cryptography jsonpointer requests stevedore idna asn1crypto chardet urllib3 certifi

johnsca commented 5 years ago

This is due in part to the isolation from the snap meaning the the python3-dev packages from the host system can't be accessed. @knkski did hit this with a different library, PyYAML, on Travis, and was able to work around it by removing libyaml-dev from the host, but this seems like a different package and that doesn't seem like an ideal general approach.

Another way you could work around this is to move the cffi library from the wheelhouse.txt to the python_packages layer option for layer:basic. This would defer fetching, installing, and building until deploy time.

I am also confused why, with the charm build, the pip download command somehow got turned into a pip install command. We specifically don't want it installing or compiling anything locally, since the generated artifacts may end up on other architectures and should thus be compiled at install time.

I briefly considered adding python3-dev to the snapcraft.yaml, but that seems like a mistake since we don't want compiled binaries embedded into the charm in any case.

dshcherb commented 5 years ago

@johnsca

https://github.com/juju/charm-tools/blob/149231f03d0d2c8458ed8e8e6208c3692f07d72d/charmtools/build/tactics.py#L946-L981

I agree that avoiding any compilation at the build machine is preferable, otherwise we are creating arch-specific charms which is what usage of python is meant to avoid.

The code in question seems to be quite old though.

Have you come to any conclusion on how to approach this problem?

afreiberger commented 5 years ago

I'm running into this issue as well when trying to add python-libmaas to wheelhouse.txt to support not only the charm's code being able to create a user within MAAS, but also a custom nagios plugin which uses that user to query status of MAAS services.

The point about not having compiled binaries within the charms makes sense unless the charm is specifically to be run on an amd64 ubuntu host running our MAAS product, for instance.

Is there some way to detect and provide warnings during charm build phase that would notify users that they're adding a compiled bit that will necessarily lock their charm into a specific architecture while allowing for the technology to be used in situations where it is warranted or safe to do so?

Using python_packages in layer:basic won't work when using venv=true in order for charm-landed scripts to function outside of the context of the charm.

In environments where they are detached from the internet, using 'pip3 install python-libmaas' will not be a suitable solution.

I will note regarding pip install vs. pip download, the latest version of charm tools does run pip download: Command failed: bash -c . /tmp/tmp_00ct2c4/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmpppxdjrnp -r /home/drew/layer-infra-node/src/wheelhouse.txt

The issue with all of these pip packages has the common thread of requiring cffi, which is the C Foreign Function Interface for Python, which necessarily needs to be compiled for the architecture(s) being supported as it allows python to interface directly with shared libraries. Any packages in wheelhouse that require this functionality are going to fail in charm building.

Oddly, there are packages python(3)-cffi-backend within the ubuntu repositories which are able to install this pre-req, but that does not get referenced as an available library during the pip(3) download as an available/pre-installed package.

I did notice that my gcc command was including some system library locations that didn't exist and I created a symlink from one of those to work around this issue:

  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/snap/charm/357/usr/include/python3.6m -I/tmp/tmphnu01ki2/include/python3.6m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.6/c/_cffi_backend.o

Since /usr/include/ffi did not exist, I hijacked this path to land a symlink to /usr/include/python3.6m which solved this for me. I will make a special note in my charm README that this is specifically intended for amd64 systems as it may break python-libmaas on other platforms.

johnsca commented 5 years ago

The point about not having compiled binaries within the charms makes sense unless the charm is specifically to be run on an amd64 ubuntu host running our MAAS product, for instance.

The issue isn't with packages requiring compilation being used in the wheelhouse, per se, because only the package source is actually bundled (which would then be compiled and installed during bootstrap on the target unit). The issue is that pip download internally uses the installation code-path to fetch and resolve dependencies which ends up trying to compile the packages within the charm snap environment, which doesn't include any of the build dependencies (and would almost certainly always be missing some for an arbitrary package). If we could avoid / prevent pip from doing the "dummy install" during pip download, then including packages requiring compilation in the wheelhouse would be a non-issue.

Another option might be to have the build process create a snapped "charm runtime" (we'd still want to keep the charm code itself outside the snap to make interactive debugging easier) and automatically include that as a resource. However, that runs up against the architecture-specific resource problem. But possibly the new charm store could handle that in a clean and graceful way.

auria commented 4 years ago

I've run into the same issue when wheelhouse.txt looks like:

psutil pylxd

I applied the same workaround mentioned by @afreiberger, which is take advantage of the included /usr/include/libffi (which didn't exist) and link it to /usr/include/python3.6

sudo ln -s /usr/include/python3.6 /usr/include/libffi

aieri commented 4 years ago

Another workaround (for openstack charms at least) is to install libffi-dev locally and build via the openstack charmers scripts instead of using the charm snap

sabaini commented 4 years ago

Another workaround (for openstack charms at least) is to install libffi-dev locally and build via the openstack charmers scripts instead of using the charm snap

Did you mean to link to https://github.com/openstack-charmers/release-tools ?

aieri commented 4 years ago

Another workaround (for openstack charms at least) is to install libffi-dev locally and build via the openstack charmers scripts instead of using the charm snap

Did you mean to link to https://github.com/openstack-charmers/release-tools ?

I did! Fixing...

juju / charm-tools