Charm cannot build-on 20.04 because of No module named 'markupsafe'

[chanchiwai-ray]

:: Setting up python3-virtualenv (20.0.17-1ubuntu0.4) ...                                                                    
:: Setting up virtualenv (20.0.17-1ubuntu0.4) ...                                                                            
:: Processing triggers for libc-bin (2.31-0ubuntu9) ...                                                                      
:: build: Destination charm directory: /root/parts/charm/build/prometheus2                                                   
:: build: Processing layer: layer:options                                                                                    
:: build: Processing layer: layer:basic                                                                                      
:: build: Processing layer: layer:snap                                                                                       
:: build: Processing layer: layer:debug                                                                                      
:: build: Processing layer: layer:tls-client                                                                                 
:: build: Processing layer: prometheus2 (from .)                                                                             
:: build: Processing interface: tls-certificates                                                                             
:: build: Processing interface: http                                                                                         
:: build: Processing interface: http                                                                                         
:: build: Processing interface: http                                                                                         
:: build: Processing interface: http                                                                                         
:: build: Processing interface: http                                                                                         
:: build: Processing interface: nrpe-external-master                                                                         
:: build: Processing interface: grafana-source                                                                               
:: build: Processing interface: prometheus                                                                                   
:: build: Processing interface: prometheus-rules                                                                             
:: build: Processing interface: prometheus-manual                                                                            
:: build: Ignoring pip: markers 'python_version < "3.8"' don't match your environment                                        
:: Ignoring Jinja2: markers 'python_version >= "3.0" and python_version <= "3.4"' don't match your environment               
:: Ignoring Jinja2: markers 'python_version == "2.7" or python_version == "3.5"' don't match your environment                
:: Ignoring PyYAML: markers 'python_version >= "3.0" and python_version <= "3.4"' don't match your environment               
:: Ignoring MarkupSafe: markers 'python_version < "3.6"' don't match your environment                                        
:: Ignoring MarkupSafe: markers 'python_version == "3.6"' don't match your environment                                       
:: Ignoring setuptools: markers 'python_version < "3.8"' don't match your environment                                        
:: Ignoring setuptools-scm: markers 'python_version < "3.8"' don't match your environment                                    
:: Ignoring wheel: markers 'python_version < "3.8"' don't match your environment                                             
:: Collecting pip<22.1                                                                                                       
::   Downloading pip-22.0.4.tar.gz (2.1 MB)                                                                                  
::   Installing build dependencies: started                                                                                  
::   Installing build dependencies: finished with status 'done'                                                              
::   Getting requirements to build wheel: started                                                                            
::   Getting requirements to build wheel: finished with status 'done'                                                        
::     Preparing wheel metadata: started                                                                                     
::     Preparing wheel metadata: finished with status 'done'                                                                 
::   Saved /root/snap/charmcraft/common/tmp1od5ul1h/pip-22.0.4.tar.gz                                                        
:: Collecting Jinja2                                                                                                         
::   Downloading Jinja2-3.1.2.tar.gz (268 kB)                                                                                
::     ERROR: Command errored out with exit status 1:                                                                        
::      command: /root/snap/charmcraft/common/tmpi8ef_m7r/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/setup.py'"'"'; __file__='"'"'/root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/pip-egg-info                                                                                             
::          cwd: /root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/                                                  
::     Complete output (37 lines):                                                                                           
::     Traceback (most recent call last):                                                                                    
::       File "<string>", line 1, in <module>                                                                                
::       File "/root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/setup.py", line 4, in <module>                      
::         setup(                                                                                                            
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup                                                                                                                            
::         return distutils.core.setup(**attrs)                                                                              
::       File "/usr/lib/python3.8/distutils/core.py", line 121, in setup                                                     
::         dist.parse_config_files()                                                                                         
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/_virtualenv.py", line 21, in parse_config_files                                                                                                                        
::         result = old_parse_config_files(self, *args, **kwargs)                                                            
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files                                                                                                                   
::         parse_configuration(self, self.command_options,                                                                   
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration                                                                                                                
::         meta.parse()                                                                                                      
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
::         section_parser_method(section_options)                                                                            
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section                                                                                                                      
::         self[name] = value                                                                                                
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__                                                                                                                        
::         value = parser(value)                                                                                             
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version                                                                                                                     
::         version = self._parse_attr(value, self.package_dir)                                                               
::       File "/root/snap/charmcraft/common/tmpi8ef_m7r/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr                                                                                                                        
::         module = import_module(module_name)                                                                               
::       File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module                                         
::         return _bootstrap._gcd_import(name[level:], package, level)                                                       
::       File "<frozen importlib._bootstrap>", line 1014, in _gcd_import                                                     
::       File "<frozen importlib._bootstrap>", line 991, in _find_and_load                                                   
::       File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked                                          
::       File "<frozen importlib._bootstrap>", line 671, in _load_unlocked                                                   
::       File "<frozen importlib._bootstrap_external>", line 848, in exec_module                                             
::       File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed                                        
::       File "/root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/src/jinja2/__init__.py", line 8, in <module>        
::         from .environment import Environment as Environment                                                               
::       File "/root/snap/charmcraft/common/pip-download-fb3bmu6d/Jinja2/src/jinja2/environment.py", line 14, in <module>    
::         from markupsafe import Markup                                                                                     
::     ModuleNotFoundError: No module named 'markupsafe'                                                                     
::     ----------------------------------------                                                                              
:: ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.           
Running step STAGE for part 'charm'                                                                                          
Execute action                                                                                                               
Running step PRIME for part 'charm'

It seems that these dependencies are comming from layer-basic. I am using charmcraft == 2.0.0.

Adding these to the charm's wheelhouse will make it build and install on 20.04, but not sure if it's the best choice.

Jinja2<3;python_version == '3.8'
setuptools~=46.0.0;python_version == '3.8'

[ajkavanagh]

So this is a curious one. layer-basic's wheelhouse.txt has the following lines:

Jinja2==2.10;python_version >= '3.0' and python_version <= '3.4' # py3 trusty
Jinja2==2.11;python_version == '2.7' or python_version == '3.5'  # py27, py35
Jinja2;python_version >= '3.6' # py36 and on

PyYAML==5.2;python_version >= '3.0' and python_version <= '3.4' # py3 trusty
PyYAML<5.4;python_version == '2.7' or python_version >= '3.5'  # all else

MarkupSafe<2.0.0;python_version < '3.6'
MarkupSafe<2.1.0;python_version == '3.6' # Just for python 3.6
MarkupSafe;python_version >= '3.7' # newer pythons

setuptools<42;python_version < '3.8'
# https://github.com/juju-solutions/layer-basic/issues/201
setuptools<62.2.0;python_version >= '3.8'
setuptools-scm<=1.17.0;python_version < '3.8'
# https://github.com/pypa/setuptools_scm/issues/722
setuptools-scm<7;python_version >= '3.8'

So the right pins are in there for building on py38 (20.04) and later (22.04 - py310). I wonder if it's the order of the MarkupSafe and Jinja2 declarations in the wheelhouse.txt?

@chanchiwai-ray do you have a link to the example layer that you are building that has this problem, please?

[chanchiwai-ray]

I was building reactive charm for example: charm-duplicity and testing build-on 20.04 and run-on 20.04, and I encounter this problem. Some other charms like charm-prometheus2 also has this problem.

The charmcraft.yaml I used for charm-duplicity was

type: charm                                                                                                                   
parts:
  charm:
    source: src
    plugin: reactive
    build-snaps: [charm]
bases:
    - build-on:
        - name: ubuntu
          channel: "20.04"
          architectures: ["amd64"]
      run-on:
        - name: ubuntu
          channel: "20.04"
          architectures:
              - amd64

[ajkavanagh]

Okay, so this isn't a bug in layer-basic, but is instead a bug in charm-tools: https://github.com/juju/charm-tools/issues/646

So keep you developing for the moment, if you change build-snaps to:

...
parts:
  ...
  build-snaps:
    - charm/2.x/stable

then the charm will build successfully. I've tested this with the duplicity charm that was linked with the 2.x/stable branch.

Unfortunately, the 3.x branch currently not compatible with building on < 22.04 (hence the bug).

I'm going to close this, but put a link to it from the charm-tools bug.