appimaged does not work on AppImages with zstd compression

[probonopd]

Hello @CalebQ42, I have updated the AppImages generated by this project to use zstd compression.

Unfortunately appimaged seems to not work on those AppImages. Can it be that zstd compression is not working in the version of the squashfs implementation that we are using here? Or is the issue something else?

Note to self:

/usr/bin/systemctl -l --no-pager --user status appimaged

shows the systemctl log.

[CalebQ42]

The version used should have support for zstd.

[probonopd]

Would appreciate if you could have a look @CalebQ42 why the AppImages from https://github.com/probonopd/go-appimage/releases/tag/continuous are not being integrated. Thanks!

[CalebQ42]

I'll look into it some more, but it seems there is something else wrong. When trying to open mkappimage and appimaged's archive, it's failing because the magic number isn't correct. It looks like somehow the offset to where the archive starts isn't being read correctly.

To be clear the appimage's work correctly, but it's not being read correctly with my library. I'll probably try with unsquashfs to see if it works.

[CalebQ42]

It seems unsquashfs has the same issue.

[CalebQ42]

Very weird, as nothing that was changed should have broken this...

Specifically I'm now getting an offset of 0 for some reason. I'll try rebuilding the AppImages to see if it's something weird with this specific build.

[probonopd]

I have also switched to a different AppImage runtime I am experimenting with (static binary, no need for libfuse2 on the system). Probably the bug is there. The AppImages seem to run, but maybe something is wrong with them due to the runtime?

[probonopd]

The magic bytes seem correct:

# Old
% hexdump /tmp/runtime-x86_64 | head -n 1
0000000 457f 464c 0102 0001 4941 0002 0000 0000

# New
% hexdump /tmp/runtime-fuse2-x86_64 | head -n 1
0000000 457f 464c 0102 0001 4941 0002 0000 0000

--appimage-offset seems to work:

% /tmp/runtime-fuse2-x86_64 --appimage-offset
553272
% ls -l /tmp/runtime-fuse2-x86_64
-rwxr-xr-x  1 user  wheel  553272  7 Mai  09:55 /tmp/runtime-fuse2-x86_64

[CalebQ42]

Weird... if the headers are correct, then the problem probably resides in goappimage and how it reads the header. But again, no changes were made that should break that...

[probonopd]

Indeed unsquashfs from the distribution doesn't extract all files...

% chmod +x ./appimaged-661-x86_64.AppImage

%  ./appimaged-661-x86_64.AppImage --appimage-offset 
553272

% unsquashfs -v
unsquashfs version 4.4 (2019/08/29)

% unsquashfs -o 553272 ./appimaged-661-x86_64.AppImage
Parallel unsquashfs: Using 2 processors
7 inodes (18 blocks) to write

[=============================================================================|     ] 17/18  94%

created 6 files
created 3 directories
created 0 symlinks
created 0 devices
created 0 fifos

% find squashfs-root 
squashfs-root
squashfs-root/AppRun
squashfs-root/usr
squashfs-root/usr/bin
squashfs-root/.DirIcon

but --appimage-extract does... what gives?

%  ./appimaged-661-x86_64.AppImage --appimage-extract
squashfs-root/.DirIcon
squashfs-root/AppRun
squashfs-root/appimage.png
squashfs-root/appimaged.desktop
squashfs-root/usr
squashfs-root/usr/bin
squashfs-root/usr/bin/appimaged
squashfs-root/usr/bin/bsdtar
squashfs-root/usr/bin/unsquashfs

Also, using unsquashfs from https://github.com/probonopd/static-tools/releases (statically linked, including squashfuse and libzstd) works correctly:

% wget https://github.com/probonopd/static-tools/releases/download/continuous/unsquashfs-x86_64
% chmod +x unsquashfs-x86_64

% ./unsquashfs-x86_64 -v       
unsquashfs version 4.5.1 (2022/03/17)

% ./unsquashfs-x86_64 -o 553272 ./appimaged-661-x86_64.AppImage

% find squashfs-root 
squashfs-root
squashfs-root/appimaged.desktop
squashfs-root/usr
squashfs-root/usr/bin
squashfs-root/usr/bin/bsdtar
squashfs-root/usr/bin/appimaged
squashfs-root/usr/bin/unsquashfs
squashfs-root/appimage.png
squashfs-root/AppRun
squashfs-root/.DirIcon

[CalebQ42]

I found the source of the issue with appimaged. It seems like the header is somehow messed up, as it isn't matching the magic number for type 2 AppImages. It's finding 410000 instead of 414902.

[CalebQ42]

❯ hexdump appimaged-655-x86_64.AppImage | head -n 1
0000000 457f 464c 0102 0001 4941 0002 0000 0000
❯ hexdump appimaged-661-x86_64.AppImage | head -n 1
0000000 457f 464c 0102 0001 0041 0000 0000 0000

[CalebQ42]

The push you just did to static-tools must have fix the issue. I just did a new build and everything is working now.

[probonopd]

Excellent. Sometimes one is blind for one's own mistakes. Seems like one must not strip the binary after inserting the magic bytes. Thanks for your help and sorry for the noise.

[CalebQ42]

No problem! I'm always happy to help, whether or not I actually end up doing anything 😋.