search

canonical / craft-parts

https://canonical-craft-parts.readthedocs-hosted.com
GNU Lesser General Public License v3.0
11 stars 36 forks source link

Stage package tracking is unaware of partitions #804

Closed mattculler closed 1 week ago

mattculler commented 1 month ago

Bug Description

If partitions are enabled and a project's parts make use of them, the LifecycleManager will try to prime files from the default prime directory, ignoring any partition prefix. The files do actually exist inside the partition's prime directory, where they are supposed to be.

Note also that the specific file that the failure occurs on will randomly be one of the primed files, it isn't always the same one. Don't know if that's relevant.

To Reproduce

Enable and make use of partitions, then prime some file(s) from stage-packages (I think?)

part yaml

kernel:
                stage-packages:
                    linux-image-generic
                organize:
                    lib/modules: (mattos-a)/lib/modules
                prime:
                    (mattos-a)/lib/modules/*/kernel/drivers/net/virtio_net.ko,
                    (mattos-a)/lib/modules/*/modules.builtin,
                    (mattos-a)/lib/modules/*/modules.builtin.modinfo,
                    (mattos-a)/lib/modules/*/modules.order,

Relevant log output

2024-08-08 13:56:15.372 execute action kernel:Action(part_name='kernel', step=Step.PRIME, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None))                                                                                                                          
2024-08-08 13:56:18.786 /home/mrc/code/willis/mattcraft/examples/coaster-os/prime/lib/modules/5.15.0-118-generic/kernel/drivers/net/virtio_net.ko: No such file or directory                                                                                                                                                                       
2024-08-08 13:56:18.786 Detailed information: FileNotFoundError: filename: '/home/mrc/code/willis/mattcraft/examples/coaster-os/prime/lib/modules/5.15.0-118-generic/kernel/drivers/net/virtio_net.ko'                                                                                                                                             
2024-08-08 13:56:18.787 Traceback (most recent call last):                                                                                                               
2024-08-08 13:56:18.787   File "/home/mrc/code/willis/mattcraft/.direnv/python-3.10.12/lib/python3.10/site-packages/craft_application/services/lifecycle.py", line 261, in run                                                                                                                                                                     
2024-08-08 13:56:18.787     aex.execute(action, stdout=stream, stderr=stream)                                                                                            
2024-08-08 13:56:18.787   File "/home/mrc/code/willis/craft-parts/craft_parts/executor/executor.py", line 327, in execute                                                
2024-08-08 13:56:18.788     self._executor.execute(actions, stdout=stdout, stderr=stderr)                                                                                
2024-08-08 13:56:18.788   File "/home/mrc/code/willis/craft-parts/craft_parts/executor/executor.py", line 137, in execute                                                
2024-08-08 13:56:18.788     self._run_action(act, stdout=stdout, stderr=stderr)                                                                                          
2024-08-08 13:56:18.788   File "/home/mrc/code/willis/craft-parts/craft_parts/executor/executor.py", line 214, in _run_action                                            
2024-08-08 13:56:18.788     handler.run_action(action, stdout=stdout, stderr=stderr)                                                                                     
2024-08-08 13:56:18.788   File "/home/mrc/code/willis/craft-parts/craft_parts/executor/part_handler.py", line 170, in run_action                                         
2024-08-08 13:56:18.788     state = handler(step_info, stdout=stdout, stderr=stderr)                                                                                     
2024-08-08 13:56:18.788   File "/home/mrc/code/willis/craft-parts/craft_parts/executor/part_handler.py", line 424, in _run_prime                                         
2024-08-08 13:56:18.788     primed_stage_packages = _get_primed_stage_packages(                                                                                          
2024-08-08 13:56:18.789   File "/home/mrc/code/willis/craft-parts/craft_parts/executor/part_handler.py", line 1140, in _get_primed_stage_packages                        
2024-08-08 13:56:18.789     stage_package = read_origin_stage_package(snap_file)                                                                                         
2024-08-08 13:56:18.789   File "/home/mrc/code/willis/craft-parts/craft_parts/packages/base.py", line 259, in read_origin_stage_package                                  
2024-08-08 13:56:18.789     return xattrs.read_xattr(path, _STAGE_PACKAGE_KEY)                                                                                           
2024-08-08 13:56:18.789   File "/home/mrc/code/willis/craft-parts/craft_parts/xattrs.py", line 47, in read_xattr                                                         
2024-08-08 13:56:18.789     value = os.getxattr(path, key)                                                                                                               
2024-08-08 13:56:18.789 FileNotFoundError: [Errno 2] No such file or directory: '/home/mrc/code/willis/mattcraft/examples/coaster-os/prime/lib/modules/5.15.0-118-generic/kernel/drivers/net/virtio_net.ko'
lengau commented 1 month ago

The fact of it not always being the same file is probably just a side-effect of prime occurring in the order in which the OS retrieves filenames from the drive.

syncronize-issues-to-jira[bot] commented 1 month ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3187.

This message was autogenerated

mattculler commented 1 month ago

When this is completed, I can switch mattcraft to use the fixed craft-parts version and then delete this branch: https://github.com/canonical/craft-parts/tree/temp-disable-stage-package-tracking

mr-cal commented 2 weeks ago

How stage packages work

pull

Fetch packages to parts/<part-name>/stage-packages

build

Unpack packages to parts/<part-name>/install Add an xattr record to each file with the name of the stage package. Organize to other partitions, if defined with an organize keyword

stage

No special handling

prime

Generate a list of all stage-packages who have primed files:

Short term solution

Collect a list of stage-packages primed in any partition.

This function should receive a dictionary of partition names to prime directories (project.get_prime_dirs) and check for stage packages per-partition:

https://github.com/canonical/craft-parts/blob/8b6d8db17605dac6ffa9fc6bcb59a39d47523e83/craft_parts/executor/part_handler.py#L1135-L1142

Long term solution

Once craft-parts supports multiple lifecycles and the StateManager is partition aware (see https://github.com/canonical/craft-parts/issues/665#issuecomment-2289510579), stage packages can be tracked per-partition.