charm pack fails with "metadata: Cannot read the metadata.yaml file."

Otherwise, if you have a feature request or issue with any other charm command please delete the above and continue below

Checklist

[X] Confirmed this is an issue with charm-tools, not charmstore-client
[x] Provide versions of tools used
[x] Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: snap info charm and got the following ouput:

installed:          2.8.2                          (609) 119MB classic

I am using: Ubuntu 22.04

Issue/Feature

Build fails as follows:

ubuntu@j1:~/designate$ tox -e build                                                                                                                                                                                                                                                                                                                                                                                                     
.tox create: /home/ubuntu/designate/.tox/.tox                                                                                                                                                                                                                                                                                                                                                                                           
.tox installdeps: pip < 20.3, virtualenv < 20.0, setuptools<50.0.0, tox >= 3.18.0                                                                                                                                                                                                                                                                                                                                                       
build create: /home/ubuntu/designate/.tox/build                                                                                                                                                                                                                                                                                                                                                                                         
build installdeps: -r/home/ubuntu/designate/build-requirements.txt                                                                                                                                                                                                                                                                                                                                                                      
build run-test-pre: PYTHONHASHSEED='0'                                                                                                                                                                                                                                                                                                                                                                                                  
build run-test: commands[0] | charmcraft clean                                                                                                                                                                                                                                                                                                                                                                                          
Cleaned project 'designate'.                                                                                                                                                                                                                                                                                                                                                                                                            
build run-test: commands[1] | charmcraft -v pack                                                                                                                                                                                                                                          
Starting charmcraft version 2.0.0                                                                                                                                                                                                                                                         
Logging execution to '/home/ubuntu/snap/charmcraft/common/cache/charmcraft/log/charmcraft-20220822-192025.667555.log'                                                                                                                                                                     
Packing the charm.                                                                                                                                                                                                                                                                        
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)                                                                                                                                          
Packing the charm                                                                                                                                                                                                                                                                         
Starting charmcraft version 2.0.0                                                                                                                                                                                                                                                         
Logging execution to '/tmp/charmcraft.log'                                                                                                                                                                                                                                                
Packing the charm.                                                                                                                                                                                                                                                                        
Building charm in '/root'                                                                                                                                                                                                                                                                 
Running step PULL for part 'charm'                                                                                                                                                                                                                                                        
Execute action                                                                                                                                                                                                                                                                            
Running step OVERLAY for part 'charm'                                                                                                                                                                                                                                                     
Execute action                                                                                                                                                                                                                                                                            
Running step BUILD for part 'charm'                                                                                                                                                                                                                                                       
Execute action                                                                                                                                                                                                                                                                            
:: + /snap/charmcraft/x1/bin/python3 -I /snap/charmcraft/x1/lib/charmcraft/reactive_plugin.py designate /root/parts/charm/build /root/parts/charm/install                                                                                                                                 
:: I: metadata name (designate) must match directory name (build) exactly for local deployment.                                                                                                                                                                                           
:: I: `display-name` not provided, add for custom naming in the UI                                                                                                                                                                                                                        
:: I: no hooks directory                                                                                                                                                                                                                                                                  
:: W: no copyright file                                                                                                                                                                                                                                                                   
:: I: relation dnsaas has no hooks                                                                                                                                                                                                                                                        
:: I: relation nrpe-external-master has no hooks                                                                                                                                                                                                                                          
:: I: relation dns-backend has no hooks                                                                                                                                                                                                                                                   
:: I: relation ha has no hooks                                                                                                                                                                                                                                                            
:: I: relation coordinator-memcached has no hooks                                                                                                                                                                                                                                         
:: I: relation cluster has no hooks                                                                                                                                                                                                                                                       
:: I: missing recommended hook install                                                                                                                                                                                                                                                    
:: I: missing recommended hook start                                                                                                                                                                                                                                                      
:: I: missing recommended hook stop                                                                                                                                                                                                                                                       
:: I: missing recommended hook config-changed                                                                                                                                                                                                                                             
:: I: config.yaml: option dns-slaves has no default value                                                                                                                                                                                                                                 
:: I: config.yaml: option nova-domain has no default value                                                                                                                                                                                                                                
:: I: config.yaml: option nova-domain-email has no default value                                                                                                                                                                                                                          
:: I: config.yaml: option nameservers has no default value                                                                                                                                                                                                                                
:: I: config.yaml: option neutron-domain has no default value                                                                                                                                                                                                                             
:: I: config.yaml: option neutron-domain-email has no default value                                                                                                                                                                                                                                                                                                        
:: I: config.yaml: option also-notifies has no default value                                                                                                                                                                                                                                                                                                               
:: I: config.yaml: option default-soa-minimum has no default value                                                                                                                                                                                                                                                                                                         
:: I: config.yaml: option default-soa-refresh-min has no default value                                                                                                                                                                                                                                                                                                     
:: I: config.yaml: option default-soa-refresh-max has no default value                                                                                                                                                                                                                                                                                                     
:: I: config.yaml: option default-soa-retry has no default value                                                                                                                                                                                                                                                                                                           
:: I: config.yaml: option default-ttl has no default value                                                                                                                                                                                                                                                                                                                 
:: I: config.yaml: option managed-resource-tenant-id has no default value                                                                                                                                                                                                                                                                                                  
:: W: config.yaml: option openstack-origin does not have the keys: description, type                                                                                                                                                                                                                                                                                                                                                    
:: build: The lockfile /root/parts/charm/build/build.lock was not found; building using latest versions.                                                                                                                                                                                                                                                                                                                                
:: build: Destination charm directory: /root/parts/charm/build/designate                                                                                                                                                                                                                                                                                                                                                                
:: build: Processing layer: layer:options                                                                                                                                                                                                                                                                                                                                                                                               
:: build: Processing layer: layer:basic                                                                                                                                                                                                                                                                                                                                                                                                 
:: build: Processing layer: layer:openstack                                                                                                                                                                                                                                                                                                                                                                                             
:: build: Processing layer: layer:openstack-principle                                                                                                                                                                                                                                                                                                                                                                                   
:: build: Processing layer: layer:openstack-api                                                                                                                                                                                                                                                                                                                                                                                         
:: build: Processing layer: layer:leadership                                                                                                                                                                                                                                                                                                                                                                                            
:: build: Processing layer: designate (from .)                                                                                                                                                                                                                                                                                                                                                                                          
:: build: Processing interface: tls-certificates                                                                                                                                                                                                                                                                                                                                                                                        
:: build: Processing interface: mysql-shared                                                                                                                                                                                                                                                                                                                                                                                            
:: build: Processing interface: rabbitmq                                                                                                                                                                                                                                                                                                                                                                                                
:: build: Processing interface: keystone                                                                                                                                                                                                                                                                                                                                                                                                
:: build: Processing interface: hacluster                                                                                                                                                                                                                                                                                                                                                                                               
:: build: Processing interface: openstack-ha                                                                                                                                                                                                                                                                                                                                                                                            
:: build: Processing interface: bind-rndc                                                                                                                                                                                                                                                                                                                                                                                               
:: build: Processing interface: memcache                                                                                                                                                                                                                                                                                                                                                                                                
:: build: Processing interface: designate (from designate)                                                                                                                                                                                                                                                                                                                                                                              
:: build: Processing interface: nrpe-external-master                                                                                                                                                                                                                                                                                                                                                                                    
:: charmtools.build.tactics: Missing implementation for interface role: provides.py                                                                                                                                                                                                                                                                                                                                                     
Running step STAGE for part 'charm'                                                                                                                                                                                                                                                                                                                                                                                                     
Execute action                                                                                                                                                                                                                                                                                                                                                                                                                          
Running step PRIME for part 'charm'                                                                                                                                                                                                                                                                                                                                                                                                     
Execute action                                                                                                                                                                                                                                                                                                                                                                                                                          
Check result: language [attribute] unknown (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyze#heading--language).                                                                                                                                                                                                                                                                              
Check result: framework [attribute] unknown (The charm is not based on any known Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyze#heading--framework).                                                                                                                                                                                                                                                               
Lint Errors:                                                                                                                                                                                                                                                                                                                                                                                                                            
- metadata: Cannot read the metadata.yaml file. (https://juju.is/docs/sdk/charmcraft-analyze#heading--metadata)                                                                                                                                                                                                                                                                                                                         
Aborting due to lint errors (use --force to override).                                                                                                                                                                                                                                                                                                                                                                                  
Failed to build charm for bases index '0'.                                                                                                                                                                                                                                                                                                                                                                                              
Full execution log: '/home/ubuntu/snap/charmcraft/common/cache/charmcraft/log/charmcraft-20220822-192025.667555.log'                                                                                                                                                                                                                                                                                                                    
ERROR: InvocationError for command /snap/bin/charmcraft -v pack (exited with code 1)                                                                                                                                
________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________                                                                                                                                              
ERROR:   build: commands failed

however, if I add the following to charmcraft.yaml, the build is successful, even though the interfaces and layers directories are empty:

ubuntu@j1:~/designate$ git diff
diff --git a/charmcraft.yaml b/charmcraft.yaml
index e8b86ef..7497a22 100644
--- a/charmcraft.yaml
+++ b/charmcraft.yaml
@@ -6,6 +6,10 @@ parts:
     plugin: reactive
     build-snaps:
       - charm
+    build-environment:
+      - CHARM_INTERFACES_DIR: /root/project/interfaces/
+      - CHARM_LAYERS_DIR: /root/project/layers/
+

 bases:
   - build-on:

To recreate:

git clone https://opendev.org/openstack/charm-designate designate
cd designate
git fetch https://review.opendev.org/openstack/charm-designate refs/changes/80/850180/11 && git checkout FETCH_HEAD
vi charmcraft.yaml # remove the build-environment mapping
tox -e build

I expect/expected the following

The problem is I'm unable to debug what I think is an interface linting issue because the metadata.yaml error goes away once I add the build-environment environment variables, even though they point to nothing.

I also had this problem, what got the build to at least work is adding the build-packages to the charmcraft.yaml file:

parts:
  charm:
    source: .
    plugin: reactive
    build-snaps: [charm]
    build-packages:
      - virtualenv
      - git

Whilst this did get the build to work, when I deploy the charm there seems to be some problem with setuptools (the following is an excerpt from the juju debug-log:

machine-1: 13:28:08 INFO juju.downloader download complete ("local:focal/jenkins-1")
machine-1: 13:28:09 INFO juju.downloader download verified ("local:focal/jenkins-1")
unit-jenkins-1: 13:28:35 INFO juju.worker.uniter hooks are retried true
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   ERROR: Command errored out with exit status 1:
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached    command: /var/lib/juju/agents/unit-jenkins-1/.venv/bin/python /var/lib/juju/agents/unit-jenkins-1/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k3p3clzn/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- setuptools wheel
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached        cwd: None
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   Complete output (39 lines):
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   Looking in links: wheelhouse
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   Processing ./wheelhouse/setuptools-65.3.0.tar.gz
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     Getting requirements to build wheel: started
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     Getting requirements to build wheel: finished with status 'error'
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     ERROR: Command errored out with exit status 1:
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached      command: /var/lib/juju/agents/unit-jenkins-1/.venv/bin/python /tmp/tmpul9p2cb4 get_requires_for_build_wheel /tmp/tmpwhc0c77q
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached          cwd: /tmp/pip-install-fx4vw08n/setuptools
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     Complete output (29 lines):
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     Traceback (most recent call last):
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/tmpul9p2cb4", line 280, in <module>
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         main()
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/tmpul9p2cb4", line 263, in main
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         json_out['return_val'] = hook(**hook_input['kwargs'])
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/tmpul9p2cb4", line 114, in get_requires_for_build_wheel
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         return hook(config_settings)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         return self._get_build_requires(config_settings, requirements=['wheel'])
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/build_meta.py", line 320, in _get_build_requires
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         self.run_setup()
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/build_meta.py", line 335, in run_setup
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         exec(code, locals())
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "<string>", line 87, in <module>
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/__init__.py", line 87, in setup
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         return distutils.core.setup(**attrs)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/_distutils/core.py", line 147, in setup
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         _setup_distribution = dist = klass(attrs)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/dist.py", line 476, in __init__
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         _Distribution.__init__(
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/_distutils/dist.py", line 280, in __init__
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         self.finalize_options()
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/dist.py", line 899, in finalize_options
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         for ep in sorted(loaded, key=by_order):
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/dist.py", line 898, in <lambda>
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         loaded = map(lambda e: e.load(), filtered)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached       File "/tmp/pip-install-fx4vw08n/setuptools/setuptools/_vendor/importlib_metadata/__init__.py", line 196, in load
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached         return functools.reduce(getattr, attrs, module)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     ----------------------------------------
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   ERROR: Command errored out with exit status 1: /var/lib/juju/agents/unit-jenkins-1/.venv/bin/python /tmp/tmpul9p2cb4 get_requires_for_build_wheel /tmp/tmpwhc0c77q Check the logs for full command output.
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   ----------------------------------------
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached ERROR: Command errored out with exit status 1: /var/lib/juju/agents/unit-jenkins-1/.venv/bin/python /var/lib/juju/agents/unit-jenkins-1/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k3p3clzn/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- setuptools wheel Check the logs for full command output.
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached Traceback (most recent call last):
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   File "/var/lib/juju/agents/unit-jenkins-1/charm/hooks/jenkins-storage-attached", line 8, in <module>
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     basic.bootstrap_charm_deps()
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   File "/var/lib/juju/agents/unit-jenkins-1/charm/lib/charms/layer/basic.py", line 202, in bootstrap_charm_deps
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     _update_if_newer(pip, pre_install_pkgs)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   File "/var/lib/juju/agents/unit-jenkins-1/charm/lib/charms/layer/basic.py", line 366, in _update_if_newer
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     check_call([pip, 'install', '-U', '--no-index', '-f', 'wheelhouse',
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached     raise CalledProcessError(retcode, cmd)
unit-jenkins-1: 13:29:48 WARNING unit.jenkins/1.jenkins-storage-attached subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-jenkins-1/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.

I think #639 has solved this issue; please could you check again, and note here if it doesn't seem to be working? Thanks!

@ajkavanagh I'm seeing the following with that change in place: https://pastebin.ubuntu.com/p/MFP3zn8pB9/

@mthaddon can you post a link to the problem charm (layer) - it would be good to see where the issue is. Thanks.

@mthaddon okay, so I got it to build by doing the following:

I changed the charmcraft.yaml to:

type: "charm"
bases:
    - build-on:
      - name: "ubuntu"
        channel: "20.04"
      run-on:
      #- name: "ubuntu"
        #channel: "16.04"
      #- name: "ubuntu"
        #channel: "18.04"
      - name: "ubuntu"
        channel: "20.04"
parts:
  charm:
    source: .
    plugin: reactive
    build-snaps:
      - charm/2.x/stable
    build-environment:
      - CHARM_INTERFACES_DIR: /root/project/interfaces/
      - CHARM_LAYERS_DIR: /root/project/layers/

Note, that a charm built on 20.04 is very unlike to run on 18.04, let alone 16.04 due to the python version differences. It's probably best to build on/run on each version separately, although we have some charms that build-on 18.04 and run on 20.04 as well. e.g.

  - build-on:
      - name: ubuntu
        channel: "18.04"
        architectures:
          - amd64
    run-on:
      - name: ubuntu
        channel: "18.04"
        architectures: [amd64, s390x, ppc64el, arm64]
      - name: ubuntu
        channel: "20.04"
        architectures: [amd64, s390x, ppc64el, arm64]

Obviously, you may not need the separate arches, in which case you'd just get amd64.

Note I had to switch the charm to the 2.x/stable channel; not sure why it won't build with 3.x at the moment.

I changed the layer.yaml to:

includes: 
  - 'layer:basic'
  - 'layer:snap'
options:
  basic:
    use_venv: True
    include_system_packages: False
repo: https://git.launchpad.net/juju-introspection-proxy-charm

This just ensures that the charm doesn't pull any packages from the distro, and is mostly self contained. Might need testing if anything in the charm was relying on system packages?

I added a blank copyright file to the layer. This seems to be a cause of the metadata.yaml file read issue!?

Anyway, hope that helps in getting it moving forward. I still think we have an issue with the 3.x snap though.

Thx for the pointers!

Note that there is an issue with the snap when running on bionic, as @esunar discovered, installing the build-packages through charmcraft as discussed here works around the issue.

The proper fix for that will most likely be in the charmcraft reactive plugin.

Not sure if this is relevant for your specific issue but thought it was worth noting since you are trying to build on bionic.

juju / charm-tools