search

canonical / snapcraft

Package, distribute, and update any app for Linux and IoT.
https://snapcraft.io
GNU General Public License v3.0
1.19k stars 448 forks source link

When cross-compiling incorrect architecture `stage-packages` are fetched #4542

Open dilyn-corner opened 10 months ago

dilyn-corner commented 10 months ago

Bug Description

When building a snap on an AMD64 machine for an ARM64 device, it was noticed by the library linter that a bunch of libraries and other files were unused -- oddly enough, they were located in /usr/lib/x86_64-linxu-gnu instead of the anticipated ARM64 triple.

This bug has been noticed on several different snaps. I have included references to those in the additional context section.

To Reproduce

git clone --branch core22-bump -d 1 https://github.com/canonical/iot-field-example-snaps
cd iot-field-example-snaps/qt-imx
snapcraft --build-for=arm64 --use-lxd

Environment

Ubuntu 23.10 running snapcraft rev. 10445 with --use-lxd

snapcraft.yaml

See the repository specified in the reproduction steps.

Relevant log output

See #4541 and add the below to the netplan snapcraft.yaml:

architectures:
  - build-on: amd64
    build-for: arm64

Do the build (it may fail, that's fine) and see what packages are in parts/netplan/stage_packages; below are the details (and some extra steps I tried to remediate, to non avail).

Building netplan for arm64
```shell root@testing-vm:~/netplan# snapcraft --destructive-mode --build-for=arm64 --verbosity=trace 2024-01-24 21:06:40.982 Starting Snapcraft 8.0.2.post11+gitbbf8ae08 2024-01-24 21:06:40.982 Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20240124-210640.975513.log' 2024-01-24 21:06:40.982 Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'verbosity': 'trace', 'version': False, 'trace': False} filtered=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:06:40.983 Using default command: 'pack' 2024-01-24 21:06:40.983 General parsed sysargs: command='pack' args=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:06:40.984 Command parsed sysargs: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:06:40.984 lifecycle command: 'pack', arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:06:40.984 command: pack, arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:06:41.002 Created build plan: build-on: amd64 build-for: arm64 2024-01-24 21:06:41.003 CPU count (from process affinity): 1 2024-01-24 21:06:41.003 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT '' 2024-01-24 21:06:41.003 Running on amd64 for arm64 2024-01-24 21:06:41.005 Initializing parts lifecycle 2024-01-24 21:06:41.005 is_snap: True, SNAP_NAME set to snapcraft 2024-01-24 21:06:41.006 process netplan:Step.PULL 2024-01-24 21:06:41.006 add action netplan:Step.PULL(ActionType.RUN) 2024-01-24 21:06:41.010 process netplan:Step.BUILD 2024-01-24 21:06:41.010 add action netplan:Step.BUILD(ActionType.RUN) 2024-01-24 21:06:41.011 process netplan:Step.STAGE 2024-01-24 21:06:41.014 add action netplan:Step.STAGE(ActionType.RUN) 2024-01-24 21:06:41.015 process netplan:Step.PRIME 2024-01-24 21:06:41.015 add action netplan:Step.PRIME(ActionType.RUN) 2024-01-24 21:06:41.015 Installing build-packages 2024-01-24 21:06:41.016 Installing build-snaps 2024-01-24 21:06:41.579 verify plugin environment for part 'netplan' 2024-01-24 21:06:41.581 Pulling netplan 2024-01-24 21:06:41.590 execute action netplan:Action(part_name='netplan', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)) 2024-01-24 21:06:41.591 Fetching stage-packages 2024-01-24 21:06:41.591 Requested stage-packages: ['libnetplan0:arm64', 'libpython3-stdlib:arm64', 'libpython3.11-minimal:arm64', 'libpython3.11-stdlib:arm64', 'netplan.io:arm64', 'python3-dbus:arm64', 'python3-minimal:arm64', 'python3-netifaces:arm64', 'python3-yaml:arm64', 'python3.11-minimal:arm64', 'python3:arm64'] 2024-01-24 21:06:41.691 Executing: ['apt-get', 'update'] 2024-01-24 21:06:42.088 :: Hit:1 http://archive.ubuntu.com/ubuntu mantic InRelease 2024-01-24 21:06:42.094 :: Hit:2 http://archive.ubuntu.com/ubuntu mantic-updates InRelease 2024-01-24 21:06:42.114 :: Hit:3 http://archive.ubuntu.com/ubuntu mantic-backports InRelease 2024-01-24 21:06:42.235 :: Hit:4 http://security.ubuntu.com/ubuntu mantic-security InRelease 2024-01-24 21:06:45.560 :: Reading package lists... 2024-01-24 21:06:46.370 Marking libpython3.11-stdlib:arm64 (and its dependencies) to be fetched 2024-01-24 21:06:46.374 Stage package not found in part 'netplan': libpython3.11-stdlib:arm64. 2024-01-24 21:06:46.384 Traceback (most recent call last): 2024-01-24 21:06:46.384 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 897, in _fetch_stage_packages 2024-01-24 21:06:46.385 fetched_packages = packages.Repository.fetch_stage_packages( 2024-01-24 21:06:46.385 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 602, in fetch_stage_packages 2024-01-24 21:06:46.385 return cls._fetch_stage_debs( # type: ignore[no-any-return] 2024-01-24 21:06:46.385 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 262, in wrapped 2024-01-24 21:06:46.385 return method(*args, **kwargs) 2024-01-24 21:06:46.385 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 647, in _fetch_stage_debs 2024-01-24 21:06:46.386 apt_cache.mark_packages(set(package_names)) 2024-01-24 21:06:46.386 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 309, in mark_packages 2024-01-24 21:06:46.386 raise errors.PackageNotFound(name_arch) 2024-01-24 21:06:46.386 craft_parts.packages.errors.PackageNotFound: Package not found: libpython3.11-stdlib:arm64. 2024-01-24 21:06:46.386 2024-01-24 21:06:46.390 During handling of the above exception, another exception occurred: 2024-01-24 21:06:46.390 Traceback (most recent call last): 2024-01-24 21:06:46.391 File "/snap/snapcraft/10445/lib/python3.10/site-packages/snapcraft/parts/parts.py", line 190, in run 2024-01-24 21:06:46.391 aex.execute(action, stdout=stream, stderr=stream) 2024-01-24 21:06:46.391 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 318, in execute 2024-01-24 21:06:46.391 self._executor.execute(actions, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.391 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 137, in execute 2024-01-24 21:06:46.392 self._run_action(act, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.392 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 205, in _run_action 2024-01-24 21:06:46.392 handler.run_action(action, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.392 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 172, in run_action 2024-01-24 21:06:46.392 state = handler(step_info, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.393 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 194, in _run_pull 2024-01-24 21:06:46.393 fetched_packages = self._fetch_stage_packages(step_info=step_info) 2024-01-24 21:06:46.393 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 905, in _fetch_stage_packages 2024-01-24 21:06:46.393 raise errors.StagePackageNotFound( 2024-01-24 21:06:46.394 craft_parts.errors.StagePackageNotFound: Stage package not found in part 'netplan': libpython3.11-stdlib:arm64. 2024-01-24 21:06:46.394 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240124-210640.975513.log' ```

Specifying all stage-packages as :${CRAFT_ARCH_BUILD_FOR}, adding the ports repositories for arm64 to /etc/apt/sources.list, and rebuilding:

Doing some extra work when building netplan for arm64
```shell root@testing-vm:~/netplan# snapcraft --destructive-mode --build-for=arm64 --enable-manifest --verbosity=trace 2024-01-24 21:09:02.870 Starting Snapcraft 8.0.2.post11+gitbbf8ae08 2024-01-24 21:09:02.870 Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20240124-210902.863075.log' 2024-01-24 21:09:02.871 Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'verbosity': 'trace', 'version': False, 'trace': False} filtered=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:09:02.871 Using default command: 'pack' 2024-01-24 21:09:02.871 General parsed sysargs: command='pack' args=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:09:02.872 Command parsed sysargs: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:09:02.872 lifecycle command: 'pack', arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:09:02.872 command: pack, arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:09:02.902 Created build plan: build-on: amd64 build-for: arm64 2024-01-24 21:09:02.903 CPU count (from process affinity): 1 2024-01-24 21:09:02.903 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT '' 2024-01-24 21:09:02.903 Running on amd64 for arm64 2024-01-24 21:09:02.905 Initializing parts lifecycle 2024-01-24 21:09:02.905 is_snap: True, SNAP_NAME set to snapcraft 2024-01-24 21:09:02.906 process netplan:Step.PULL 2024-01-24 21:09:02.907 add action netplan:Step.PULL(ActionType.RUN) 2024-01-24 21:09:02.907 process netplan:Step.BUILD 2024-01-24 21:09:02.907 add action netplan:Step.BUILD(ActionType.RUN) 2024-01-24 21:09:02.908 process netplan:Step.STAGE 2024-01-24 21:09:02.908 add action netplan:Step.STAGE(ActionType.RUN) 2024-01-24 21:09:02.908 process netplan:Step.PRIME 2024-01-24 21:09:02.910 add action netplan:Step.PRIME(ActionType.RUN) 2024-01-24 21:09:02.911 Installing build-packages 2024-01-24 21:09:02.911 Installing build-snaps 2024-01-24 21:09:03.470 verify plugin environment for part 'netplan' 2024-01-24 21:09:03.470 Pulling netplan 2024-01-24 21:09:03.478 execute action netplan:Action(part_name='netplan', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)) 2024-01-24 21:09:03.478 remove directory /root/netplan/parts/netplan/src 2024-01-24 21:09:03.479 Fetching stage-packages 2024-01-24 21:09:03.479 Requested stage-packages: ['libnetplan0:arm64', 'libpython3-stdlib:arm64', 'libpython3.11-minimal:arm64', 'libpython3.11-stdlib:arm64', 'netplan.io:arm64', 'python3-dbus:arm64', 'python3-minimal:arm64', 'python3-netifaces:arm64', 'python3-yaml:arm64', 'python3.11-minimal:arm64', 'python3:arm64'] 2024-01-24 21:09:03.581 Executing: ['apt-get', 'update'] 2024-01-24 21:09:04.160 :: Hit:1 http://archive.ubuntu.com/ubuntu mantic InRelease 2024-01-24 21:09:04.160 :: Hit:2 http://security.ubuntu.com/ubuntu mantic-security InRelease 2024-01-24 21:09:04.178 :: Hit:3 http://ports.ubuntu.com/ubuntu-ports mantic InRelease 2024-01-24 21:09:04.225 :: Hit:4 http://archive.ubuntu.com/ubuntu mantic-updates InRelease 2024-01-24 21:09:04.275 :: Hit:5 http://ports.ubuntu.com/ubuntu-ports mantic-updates InRelease 2024-01-24 21:09:04.322 :: Hit:6 http://archive.ubuntu.com/ubuntu mantic-backports InRelease 2024-01-24 21:09:04.369 :: Hit:7 http://ports.ubuntu.com/ubuntu-ports mantic-backports InRelease 2024-01-24 21:09:04.462 :: Hit:8 http://ports.ubuntu.com/ubuntu-ports mantic-security InRelease 2024-01-24 21:09:13.174 :: Reading package lists... 2024-01-24 21:09:22.516 Marking libpython3.11-minimal:arm64 (and its dependencies) to be fetched 2024-01-24 21:09:22.518 package: libpython3.11-minimal:arm64 2024-01-24 21:09:22.998 Marking libpython3.11-stdlib:arm64 (and its dependencies) to be fetched 2024-01-24 21:09:22.999 package: libpython3.11-stdlib:arm64 2024-01-24 21:09:23.510 Marking netplan.io:arm64 (and its dependencies) to be fetched 2024-01-24 21:09:23.511 package: netplan.io:arm64 2024-01-24 21:09:23.714 Stage package not found in part 'netplan': netplan.io:arm64. 2024-01-24 21:09:23.720 Traceback (most recent call last): 2024-01-24 21:09:23.725 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 897, in _fetch_stage_packages 2024-01-24 21:09:23.726 fetched_packages = packages.Repository.fetch_stage_packages( 2024-01-24 21:09:23.726 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 602, in fetch_stage_packages 2024-01-24 21:09:23.726 return cls._fetch_stage_debs( # type: ignore[no-any-return] 2024-01-24 21:09:23.726 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 262, in wrapped 2024-01-24 21:09:23.726 return method(*args, **kwargs) 2024-01-24 21:09:23.726 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 647, in _fetch_stage_debs 2024-01-24 21:09:23.726 apt_cache.mark_packages(set(package_names)) 2024-01-24 21:09:23.727 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 330, in mark_packages 2024-01-24 21:09:23.727 _verify_marked_install(package) 2024-01-24 21:09:23.727 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 377, in _verify_marked_install 2024-01-24 21:09:23.727 raise errors.PackageNotFound(package.name) 2024-01-24 21:09:23.727 craft_parts.packages.errors.PackageNotFound: Package not found: netplan.io:arm64. 2024-01-24 21:09:23.727 2024-01-24 21:09:23.727 During handling of the above exception, another exception occurred: 2024-01-24 21:09:23.727 Traceback (most recent call last): 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/snapcraft/parts/parts.py", line 190, in run 2024-01-24 21:09:23.728 aex.execute(action, stdout=stream, stderr=stream) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 318, in execute 2024-01-24 21:09:23.728 self._executor.execute(actions, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 137, in execute 2024-01-24 21:09:23.728 self._run_action(act, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 205, in _run_action 2024-01-24 21:09:23.728 handler.run_action(action, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 172, in run_action 2024-01-24 21:09:23.729 state = handler(step_info, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.729 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 194, in _run_pull 2024-01-24 21:09:23.729 fetched_packages = self._fetch_stage_packages(step_info=step_info) 2024-01-24 21:09:23.734 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 905, in _fetch_stage_packages 2024-01-24 21:09:23.734 raise errors.StagePackageNotFound( 2024-01-24 21:09:23.734 craft_parts.errors.StagePackageNotFound: Stage package not found in part 'netplan': netplan.io:arm64. 2024-01-24 21:09:23.734 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240124-210902.863075.log' ```

Additional context

This has been discussed for a long while, I have two reproduction cases and some chat history to share.

An example and some additional information about what I've discovered thus far can be found in this PR -- to replicate, build the qt-imx snap on the core22-bump branch. I have additionally attached a --verbosity=trace log of an example build of this snap to this issue. Given that log file, I suspect that the bug is actually more a problem with apt than with snapcraft. However, I am not certain. snapcraft-20231213-114904.997147.log.tar.gz There was an earlier discussion of this issue on the Charmhub snapcraft channel here.

There are also the following bug reports on Launchpad (since 2020):

1881515 1921418 2012503 2018609

For another example where this was noticed, see issue #4541

dilyn-corner commented 10 months ago

In at least the netplan case, adding the below results in a correct snap.

 package-repositories:
   - type: apt
     components: [main, universe]
     architectures: [arm64, armhf]
     suites: [jammy, jammy-security, jammy-updates]
     key-id: F6ECB3762474EDA9D21B7022871920D1991BC93C
     url: http://ports.ubuntu.com/ubuntu-ports

Though I do have to explicitly set:

stage-packages:
  - foo:${CRAFT_ARCH_BUILD_FOR}

For each package, otherwise I get all amd64 libraries instead of arm64 ones.

Note that while this resolves the issue in the netplan example, it does NOT resolve the issue with my qt-imx snap.

dilyn-corner commented 10 months ago

Also note that I see different behavior when setting:

base: core24
build-base: devel

And including the noble PPA as in the previous comment (s/jammy/noble).

dilyn@Ares:~/work/snaps/reproduction/netplan24 -> snapcraft clean; snapcraft --build-for=arm64 --verbosity=verbose --debug 
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20240205-185601.505656.log'                                                             
Running on amd64 for arm64                                                                                                                                     
The 'snap' directory is meant specifically for snapcraft, but it contains
the following non-snapcraft-related paths:
- snapcraft.yaml.bkp

This is unsupported and may cause unexpected behavior. If you must store
these files within the 'snap' directory, move them to 'snap/local'
which is ignored by snapcraft.          
Cleaning build provider                                                                                                                                        
Cleaned build provider                                                                                                                                         
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20240205-185603.008000.log'                                                             
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20240205-185603.008000.log'                                                             
Running on amd64 for arm64                                                                                                                                     
The 'snap' directory is meant specifically for snapcraft, but it contains
the following non-snapcraft-related paths:
- snapcraft.yaml.bkp

This is unsupported and may cause unexpected behavior. If you must store
these files within the 'snap' directory, move them to 'snap/local'
which is ignored by snapcraft.          
Running snapcraft with a devel instance is for testing purposes only.                                                                                          
Launching instance...                                                                                                                                          
Creating new instance from remote                                                                                                                              
Creating new base instance from remote                                                                                                                         
Creating new instance from base instance                                                                                                                       
Starting instance                                                                                                                                              
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/tmp/snapcraft.log'                                                                                                                      
Running on amd64 for arm64                                                                                                                                     
Initializing parts lifecycle                                                                                                                                   
Installing package repositories...                                                                                                                             
Refreshing package repositories...                                                                                                                             
Installed package repositories                                                                                                                                 
Installing build-packages                                                                                                                                      
Installing build-snaps                                                                                                                                         
Pulling netplan                                                                                                                                                
Fetching stage-packages                                                                                                                                        
Stage package not found in part 'netplan': netplan.io:arm64.                                                                                                   
Launching shell on build environment...                                                                                                                        
snapcraft-netplan-test-on-amd64-for-arm64-1415688 ../project# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease         
Hit:3 http://archive.ubuntu.com/ubuntu noble-backports InRelease                      
Hit:4 http://security.ubuntu.com/ubuntu noble-security InRelease                      
Hit:5 http://ports.ubuntu.com/ubuntu-ports noble InRelease
Hit:6 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease
Hit:7 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
22 packages can be upgraded. Run 'apt list --upgradable' to see them.
snapcraft-netplan-test-on-amd64-for-arm64-1415688 ../project# apt search netplan.io
Sorting... Done
Full Text Search... Done
netplan.io/noble 0.107.1-3 arm64
  YAML network configuration abstraction for various backends

snapcraft-netplan-test-on-amd64-for-arm64-1415688 ../project# apt search netplan.io:arm64
Sorting... Done
Full Text Search... Done
snapcraft-netplan-test-on-amd64-for-arm64-1415688 ../project# exit
exit
Stage package not found in part 'netplan': netplan.io:arm64.                                                                                                   
Failed to execute pack in instance.                                                                                                                            
Recommended resolution: Run the same command again with --debug to shell into the environment if you wish to introspect this failure.                          
Full execution log: '/home/dilyn/.local/state/snapcraft/log/snapcraft-20240205-185603.008000.log'                                                              
dilyn@Ares:~/work/snaps/reproduction/netplan24 -> snapcraft --build-for=arm64 --verbosity=verbose --debug                  
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20240205-190157.433630.log'                                                             
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20240205-190157.433630.log'                                                             
Running on amd64 for arm64                                                                                                                                     
The 'snap' directory is meant specifically for snapcraft, but it contains
the following non-snapcraft-related paths:
- snapcraft.yaml.bkp

This is unsupported and may cause unexpected behavior. If you must store
these files within the 'snap' directory, move them to 'snap/local'
which is ignored by snapcraft.          
Running snapcraft with a devel instance is for testing purposes only.                                                                                          
Launching instance...                                                                                                                                          
Starting instance                                                                                                                                              
Starting Snapcraft 8.0.2.post14+git89961cfc                                                                                                                    
Logging execution to '/tmp/snapcraft.log'                                                                                                                      
Running on amd64 for arm64                                                                                                                                     
Initializing parts lifecycle                                                                                                                                   
Installing package repositories...                                                                                                                             
Installed package repositories                                                                                                                                 
Installing build-packages                                                                                                                                      
Installing build-snaps                                                                                                                                         
Pulling netplan                                                                                                                                                
Fetching stage-packages                                                                                                                                        
Stage package not found in part 'netplan': netplan.io:arm64.                                                                                                   
Launching shell on build environment...

The package netplan.io IS found by apt (indeed, it only finds netplan.io:arm64, but cannot find this package when I specifically search for it...), but cannot be staged. for some reason.

They do indeed get marked, however!

2024-02-05 19:08:15.508 execute action netplan:Action(part_name='netplan', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None))                                                                                                     
2024-02-05 19:08:15.508 remove directory /root/parts/netplan/src                                                                                               
2024-02-05 19:08:15.509 Fetching stage-packages                                                                                                                
2024-02-05 19:08:15.509 Requested stage-packages: ['libnetplan0:arm64', 'libpython3-stdlib:arm64', 'libpython3.11-minimal:arm64', 'libpython3.11-stdlib:arm64', 'netplan.io:arm64', 'python3-dbus:arm64', 'python3-minimal:arm64', 'python3-netifaces:arm64', 'python3-yaml:arm64', 'python3.11-minimal:arm64', 'python3:arm64']                                                                                                                                                               
2024-02-05 19:08:15.547 Executing: ['apt-get', 'update']                                                                                                       
2024-02-05 19:08:15.753 :: Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease                                                                              
2024-02-05 19:08:15.760 :: Hit:2 http://security.ubuntu.com/ubuntu noble-security InRelease                                                                    
2024-02-05 19:08:15.775 :: Hit:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease                                                                      
2024-02-05 19:08:15.803 :: Hit:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease                                                                    
2024-02-05 19:08:15.869 :: Hit:5 http://ports.ubuntu.com/ubuntu-ports noble InRelease                                                                          
2024-02-05 19:08:15.964 :: Hit:6 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease                                                                 
2024-02-05 19:08:16.059 :: Hit:7 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease                                                                  
2024-02-05 19:08:18.228 :: Reading package lists...                                                                                                            
2024-02-05 19:08:19.103 Marking python3-dbus:arm64 (and its dependencies) to be fetched                                                                        
2024-02-05 19:08:19.104 package: python3-dbus:arm64                                                                                                            
2024-02-05 19:08:19.202 Marking python3-yaml:arm64 (and its dependencies) to be fetched                                                                        
2024-02-05 19:08:19.202 package: python3-yaml:arm64                                                                                                            
2024-02-05 19:08:19.260 Marking libpython3-stdlib:arm64 (and its dependencies) to be fetched                                                                   
2024-02-05 19:08:19.260 package: libpython3-stdlib:arm64                                                                                                       
2024-02-05 19:08:19.292 Marking netplan.io:arm64 (and its dependencies) to be fetched                                                                          
2024-02-05 19:08:19.292 package: netplan.io:arm64                                                                                                              
2024-02-05 19:08:19.323 Stage package not found in part 'netplan': netplan.io:arm64.                                                                           
2024-02-05 19:08:19.329 Traceback (most recent call last):                                                                                                     
2024-02-05 19:08:19.329   File "/snap/snapcraft/10495/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 897, in _fetch_stage_packages   
2024-02-05 19:08:19.329     fetched_packages = packages.Repository.fetch_stage_packages(                                                                       
2024-02-05 19:08:19.329   File "/snap/snapcraft/10495/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 602, in fetch_stage_packages             
2024-02-05 19:08:19.329     return cls._fetch_stage_debs(  # type: ignore[no-any-return]                                                                       
2024-02-05 19:08:19.329   File "/snap/snapcraft/10495/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 262, in wrapped                          
2024-02-05 19:08:19.329     return method(*args, **kwargs)                                                                                                     
2024-02-05 19:08:19.329   File "/snap/snapcraft/10495/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 647, in _fetch_stage_debs                
2024-02-05 19:08:19.329     apt_cache.mark_packages(set(package_names))                                                                                        
2024-02-05 19:08:19.329   File "/snap/snapcraft/10495/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 330, in mark_packages              
2024-02-05 19:08:19.329     _verify_marked_install(package)                                                                                                    
2024-02-05 19:08:19.329   File "/snap/snapcraft/10495/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 377, in _verify_marked_install     
2024-02-05 19:08:19.329     raise errors.PackageNotFound(package.name)                                                                                         
2024-02-05 19:08:19.329 craft_parts.packages.errors.PackageNotFound: Package not found: netplan.io:arm64.

The relevant snapcraft.yaml:

name: netplan-test
version: "0.2"
summary: Netplan
description: |
  Netplan

confinement: strict
type: app
# base: core22
base: core24
build-base: devel

grade: devel
# grade: stable

architectures:
  - build-on: amd64
    build-for: arm64

package-repositories:
  - type: apt
    components: [main, universe]
    architectures: [arm64, armhf]
    suites: [noble, noble-security, noble-updates]
    key-id: F6ECB3762474EDA9D21B7022871920D1991BC93C
    url: http://ports.ubuntu.com/ubuntu-ports

parts:
  netplan:
    plugin: nil
    stage-packages:
      - libnetplan0:${CRAFT_ARCH_BUILD_FOR}
      - libpython3-stdlib:${CRAFT_ARCH_BUILD_FOR}
      - libpython3.11-minimal:${CRAFT_ARCH_BUILD_FOR}
      - libpython3.11-stdlib:${CRAFT_ARCH_BUILD_FOR}
      - netplan.io:${CRAFT_ARCH_BUILD_FOR}
      - python3.11-minimal:${CRAFT_ARCH_BUILD_FOR}
      - python3-minimal:${CRAFT_ARCH_BUILD_FOR}
      - python3:${CRAFT_ARCH_BUILD_FOR}
      - python3-dbus:${CRAFT_ARCH_BUILD_FOR}
      - python3-netifaces:${CRAFT_ARCH_BUILD_FOR}
      - python3-yaml:${CRAFT_ARCH_BUILD_FOR}
lengau commented 9 months ago

For the piece about having to add the target architecture, I believe it's because of this line in craft-parts. I believe the fix is:

  1. Make that line point to the target architecture.
  2. if we're cross-compiling add the target architecture's repositories during the repository handling stage.
dilyn-corner commented 8 months ago

I have another reproducible use-case for this bug.

You can find the repository here. I have left some comments in the YAML explaining what happens for each case; for instance here.

I have likewise attached the entire debug log of a build from AMD64 to RISC-V64. Of note from the log: the packages which were fetched for the wrong architecture have changed from when I first attempted building this snap a few weeks ago! The list used to be far longer than just the two packages the log indicates. snapcraft-20240315-184358.020430.log Indeed, the list is so different it has actually broken my snap; the incorrect mawk was fetched and now I have an AMD64 mawk in my snap!

root@noble-vm:~/base-snap# find prime/ -name mawk -exec file {} \;
prime/usr/bin/mawk: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4ba30c5f6bc87fa93539ddc5bfd708ab51747db5, for GNU/Linux 3.2.0, stripped

I'm a big +1 on Alex's suggestion, maybe I'll take a wack at it...

syncronize-issues-to-jira[bot] commented 8 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-2620.

This message was autogenerated