search

canonical / snapcraft

Package, distribute, and update any app for Linux and IoT.
https://snapcraft.io
GNU General Public License v3.0
1.18k stars 447 forks source link

push of locally built snap fails on arm64 device #5132

Open kenvandine opened 1 month ago

kenvandine commented 1 month ago

Bug Description

When attempting to push a locally built snap to the store with my qualcomm based laptop I get the following stderr:

FATAL ERROR: Data queue size is too large

To Reproduce

snapcraft push SOMELOCALLYBUILT.snap

Environment

snapcraft: 12827

uname: Linux xps13-qc 6.11.0-35-qcom-x1e #35-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 23 15:18:06 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

snapcraft.yaml

NA

Relevant log output

2024-10-22 16:18:30.743 Starting snapcraft, version 8.4.3
2024-10-22 16:18:30.743 Log verbosity level set to BRIEF
2024-10-22 16:18:30.743 Preparing application...
2024-10-22 16:18:30.744 Configuring application...
2024-10-22 16:18:30.744 Setting up ConfigService
2024-10-22 16:18:30.758 Build plan: platform=None, build_for=None
2024-10-22 16:18:30.758 Running snapcraft push on host
2024-10-22 16:18:30.819 snapcraft internal error: SnapDataExtractionError('thunderbird_128.3.3esr-1_arm64.snap')
2024-10-22 16:18:30.829 Traceback (most recent call last):
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft_legacy/_store.py", line 58, in get_data_from_snap_file
2024-10-22 16:18:30.829     output = subprocess.check_output(
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/usr/lib/python3.10/subprocess.py", line 421, in check_output
2024-10-22 16:18:30.829     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/usr/lib/python3.10/subprocess.py", line 526, in run
2024-10-22 16:18:30.829     raise CalledProcessError(retcode, process.args,
2024-10-22 16:18:30.829 subprocess.CalledProcessError: Command '['/snap/snapcraft/12827/usr/bin/unsquashfs', '-d', '/tmp/tmpoh95i6tt/squashfs-root', PosixPath('thunderbird_128.3.3esr-1_arm64.snap'), 'meta/snap.yaml', 'snap/manifest.yaml']' returned non-zero exit status 1.
2024-10-22 16:18:30.829 
2024-10-22 16:18:30.829 During handling of the above exception, another exception occurred:
2024-10-22 16:18:30.829 Traceback (most recent call last):
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/craft_application/application.py", line 568, in run
2024-10-22 16:18:30.829     return_code = self._run_inner()
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft/application.py", line 215, in _run_inner
2024-10-22 16:18:30.829     return_code = super()._run_inner()
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/craft_application/application.py", line 549, in _run_inner
2024-10-22 16:18:30.829     return_code = dispatcher.run() or os.EX_OK
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
2024-10-22 16:18:30.829     return self._loaded_command.run(self._parsed_command_args)
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft/commands/upload.py", line 116, in run
2024-10-22 16:18:30.829     snap_yaml, manifest_yaml = get_data_from_snap_file(snap_file)
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft_legacy/_store.py", line 70, in get_data_from_snap_file
2024-10-22 16:18:30.829     raise SnapDataExtractionError(os.path.basename(snap_path))
2024-10-22 16:18:30.829 snapcraft_legacy.internal.errors.SnapDataExtractionError: Cannot read data from snap 'thunderbird_128.3.3esr-1_arm64.snap'. The file may be corrupted.


### Additional context

_No response_
syncronize-issues-to-jira[bot] commented 1 month ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3614.

This message was autogenerated

mr-cal commented 1 month ago

I was unable to reproduce this on an M1.

@kenvandine was unable to reproduce this when running the same subprocess call manually /snap/snapcraft/current/usr/bin/unsquashfs -d /tmp/tmpoh95i6tt/squashfs-root SOMELOCALLYBUILT.snap meta/snap.yaml snap/manifest.yaml on the same machine. It could be useful to try snap run --shell snapcraft and then try unsquashing. Perhaps it's something in snapd's environment.

The snap/manifest.yaml is a red herring - unsquashfs should not fail if the file does not exist.

The FATAL ERROR: Data queue size is too large error comes from unsquashfs: https://github.com/plougher/squashfs-tools/blob/0d1e6b0944f3938962238d886d9d1a8bfdf4dfcc/squashfs-tools/unsquashfs.c#L4607-L4608

Snapcraft is also using the version of squashfs-tools that shipped with Jammy. There have been many bugfixes since then, so it would be instructive to build and test with a copy of snapcraft that uses a newer version.

dariuszd21 commented 2 weeks ago

What's the size of your tmpfs ? As I can see you're on XPS with X1 chip, so you're probably using 24.10. May it be related with the oracular change that /tmp is tmpfs mount ? Out of curiosity, could you check the size of your /tmp dir?

df -h /tmp

https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878

kenvandine commented 2 weeks ago

What's the size of your tmpfs ? As I can see you're on XPS with X1 chip, so you're probably using 24.10. May it be related with the oracular change that /tmp is tmpfs mount ? Out of curiosity, could you check the size of your /tmp dir?

df -h /tmp

https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878

You are correct, I am on 24.10 and it is using tmpfs for /tmp

df reports /tmp is 15G with 15G available

dariuszd21 commented 1 week ago

Out of curiosity what's the ulimit -n result on your laptop, I found an interesting issue that may be causing this kind of problems

kenvandine commented 1 week ago

Out of curiosity what's the ulimit -n result on your laptop, I found an interesting issue that may be causing this kind of problems

ulimit is 122376

sergio-costas commented 1 week ago

I had a similar error too: https://forum.snapcraft.io/t/fatal-error-data-queue-size-is-too-large/43875

I did a test with this little program:

#!/usr/bin/env python3

import subprocess
import sys
import os
from pathlib import Path

temp_dir = "."
snap_path = sys.argv[1]

output = subprocess.check_output(
    [
        "/snap/snapcraft/current/usr/bin/unsquashfs",
        "-d",
        os.path.join(temp_dir, "squashfs-root"),
        snap_path,
        # cygwin unsquashfs on windows uses unix paths.
        Path("meta", "snap.yaml").as_posix(),
        Path("snap", "manifest.yaml").as_posix(),
    ]
)
print(output)

and it fails. But if I replace /snap/snapcraft/current/usr/bin/unsquashfs with /usr/bin/unsquashfs, it does work.

The snap has unsquashfs version 4.5, while my system has version 4.6.1.

EDIT: my test was done in an AMD x86_64 laptop.

sergio-costas commented 1 week ago

BTW: maybe the size of the snap has something to do. Mine is 1,7GB (yes, yes, I know...)

sergio-costas commented 1 week ago

Hey @kenvandine , can you test this patch when you have some spare time, please? It should fix your problem: https://github.com/canonical/snapcraft/pull/5147

It fixes my problem, at least...

kenvandine commented 1 week ago

Hey @kenvandine , can you test this patch when you have some spare time, please? It should fix your problem: #5147

It fixes my problem, at least...

I can confirm it does fix the problem I ran into on my arm64 device.

sergio-costas commented 1 week ago

@kenvandine Thanks!!!