AppImage type detection wrong

probonopd commented 3 months ago

As per the AppImageSpec, currently there are only two image formats: Types 1 and 2.

Type 1 can be identified by checking for the magic hex 0x414901 at offset 8.
Type 2 can be identified by checking for the magic hex 0x414902 at offset 8.

So why is this talking about a type 3 then? And why is it grepping for the string AppImages require FUSE to run? You should read the AppImageSpec more, instead of just making assumptions that may or may not happen to be true sometimes.

https://github.com/ivan-hc/AM/blob/6f29f8ee059a55eaa66544f928d0817205ec4e0e/modules/files.am#L68-L82

To get the image type, you might use something like this (untested):

#!/bin/sh

get_appimage_type() {
  local file_path=$1
  local content_type=$(file --mime-type -b "$file_path")

  # Check the magic bytes to determine the AppImage type
  if [ -f "$file_path" ]; then
    magic=$(dd bs=1 skip=8 count=4 if="$file_path" 2>/dev/null)
    case $magic in
      $'\x41\x49\x01') echo 1; return;;
      $'\x41\x49\x02') echo 2; return;;
    esac
  fi

  # If we couldn't determine the type, return 0 as default
  echo 0
}

https://github.com/mijorus/gearlever/issues/132

Samueru-sama commented 3 months ago

In this case appimage2 and appimage3 indicate the libfuse version they are compatible with and not the type, appimage3 means that it is using the static runtime which lets it run on fuse3 systems.

The string AppImages require FUSE to run is something that is only present on the static runtime.

probonopd commented 3 months ago

If you tell me what you want to check then I can tell you a proper way for how to check it.

In this case appimage2 and appimage3 indicate the libfuse version they are compatible with

I highly doubt this, since there are no AppImages that are "compatible with libfuse3". These AppImages statically link libfuse, so that no libfuse is needed on the system anymore at all. With the static runtime, it does not matter to the user anymore whether libfuse is installed at all or which version of it, since the static runtime comes with its own.

The string AppImages require FUSE to run is something that is only present on the static runtime.

That is pure coincidence and I don't even know whether it is always the case. Don't rely on it. If you want to check whether you have the static runtime, then check whether the file is a static binary.

Samueru-sama commented 3 months ago

I highly doubt this, since there are no AppImages that are "compatible with libfuse3". These AppImages statically link libfuse, so that no libfuse is needed on the system anymore at all. With the static runtime, it does not matter to the user anymore whether libfuse is installed at all or which version of it, since the static runtime comes with its own.

When I say fuse3 systems, that is systems that only have fusermount3 installed. Which only the appimages with the static runtime can work with.

That is pure coincidence and I don't even know whether it is always the case. Don't rely on it. If you want to check whether you have the static runtime, then check whether the file is a static binary.

So far it has always been the case, and using ldd on the appimage always says that it is a static binary, even for the appimages that rely on libfuse2.

Here an example with librewolf:

Also I want to say that I made a mistake in my first comment, I said The string AppImages require FUSE to run is something that is only present on the static runtime. when it is something NOT present in the static runtime.

probonopd commented 3 months ago

Edit: I was likely wrong here, see below.

When I say fuse3 systems, that is systems that only have fusermount3 installed. Which only the appimages with the static runtime can work with.

This is a very interesting aspect you are bringing up. People have claimed that all distributions that ship fusermount3 also symlink it as fusermount. Do you have a concrete example where this is NOT the case?

In any case, if the symlink is missing, then your tool could instruct users to create the missing symlink.

Samueru-sama commented 3 months ago

People have claimed that all distributions that ship fusermount3 also symlink it as fusermount. Do you have a concrete example where this is NOT the case?

In any case, if the symlink is missing, then your tool could instruct users to create the missing symlink.

That's true, but that's still a fusermount3 binary only, having a symlink named fusermount doesn't make it less true, the appimage that depends on libfuse2 will still fail to work.

probonopd commented 3 months ago

Are you just thinking this or have you tried it?

Samueru-sama commented 3 months ago

Are you just thinking this or have you tried it?

I've tried it, and I just did a test right above showing it?

probonopd commented 3 months ago

No, you showed above that a type-2 AppImage that does not use a static runtime requires libfuse2 to run. This has nothing to do with fusermount*.

Samueru-sama commented 3 months ago

No, you showed above that a type-2 AppImage that does not use a static runtime requires libfuse2 to run. This has nothing to do with fusermount*.

Are you saying that it is possible for a distro to ship fusermount3 only (with a symlink to fusermount) and libfuse2 alongside?

probonopd commented 3 months ago

FUSE was designed so that libfuse2 (fusermount) and libfuse3 (and with the latter, fusermount3) can be installed alongside each other. If I think about it, this speaks against the theory that there is a symlink from fusermount3 to fusermount or vice versa. Although I have not looked at every single distribution to see what they are doing.

Samueru-sama commented 3 months ago

FUSE was designed so that libfuse2 (fusermount) and libfuse3 (and with the latter, fusermount3) can be installed alongside each other. If I think about it, this speaks against the theory that there is a symlink from fusermount3 to fusermount or vice versa. Although I have not looked at every single distribution to see what they are doing.

I know fuse2 and fuse3 can be installed alongside, that's how arch handles it.

However ubuntu and maybe fedora as well, this isn't possible, so in practice we end up that distros that have fusermount3 only (with the symlink to fusermount of course) and while fuse2 might still be a separate package, it cannot be installed alongside, that means that only the appimages with the static runtime can run on fuse3 distros, hence the appimage2 vs appimage3 in the description of the appimage.

I know the issue is only that the libfuse2 library is missing, AM actually has a way to fix this as well by directly copying the libfuse2 library from the deb package into the system, I'm not a fan of that solution and I've talk with Ivan that it could cause problems but it seems to work nonetheless.

I have not looked at every single distribution to see what they are doing.

It is pain, we've had issues with wget because fedora also recently pulled this move which not even the wget2 devs agreed with.

probonopd commented 3 months ago

Then for the avoidance of confusion, please mark them as "needs libfuse2 and fusermount" vs. "does not need libfuse but needs fusermount3 or any other version", something like that. To make clear what you actually mean.

And if Fedora makes stuff cumbersome, then please do open bug requests with them, describing why their moves are making your life miserable. So that at least they know.

Thanks!

Samueru-sama commented 3 months ago

Then for the avoidance of confusion, please mark them as "needs libfuse2 and fusermount" vs. "does not need libfuse but needs fusermount3 or any other version", something like that. To make clear what you actually mean.

We could add a description in the footer of am -fthat says that.

And if Fedora makes stuff cumbersome, then please do open bug requests with them, describing why their moves are making your life miserable. So that at least they know.

We ended up working around the issue, since in the wget2 will eventually replace wget in the end, better fix it now instead.

However you need to aware of that, this example on how to use go-appimage

wget -q https://github.com/probonopd/go-appimage/releases/expanded_assets/continuous -O - | grep "appimagetool-.*-x86_64.AppImage" | head -n 1 | cut -d '"' -f 2

That will not work with wget2 which is the wget in fedora, and it isn't really a blame of the wget2 devs or even fedora, it is just that the different user agent of wget2 causes github to send the json ugly and that will make the grep and cut fail 😅.

probonopd commented 3 months ago

I stopped caring about Fedora and anything that comes out of IBM Red Hat a long time ago. They seem to make everything change as often as possible, and make everything as complicated and cumbersome as they can think of. Which is kind of logical if you get your income from support services.

ivan-hc commented 3 months ago

I stopped caring about Fedora and anything that comes out of IBM Red Hat a long time ago. They seem to make everything change as often as possible, and make everything as complicated and cumbersome as they can think of. Which is kind of logical if you get your income from support services.

Unfortunately, anything that passes for Fedora, sooner or later becomes a "de facto standard" in Linux.

Sad but true.

I test also on a VM of Fedora because of this. Since they changed wget with a symlink to wget2

ivan-hc commented 3 months ago

@probonopd just added this message in the dev branch of this repo

Istantanea_2024-08-08_01-46-56

but as @Samueru-sama suggested to me, your function in the first comment, it does not determine if a binary is static or not.

probonopd commented 3 months ago

Please, if you like the AppImage format, do not call these things "appimage2" and "appimage3". You could write "Note 1" and "Note 2" or something like that.

probonopd commented 3 months ago

By the way, I hope that we can make the type2-runtime work with any version of fusermount (unless they break it in incompatible ways).

ivan-hc commented 3 months ago

Please, if you like the AppImage format, do not call these things "appimage2" and "appimage3". You could write "Note 1" and "Note 2" or something like that.

have you got a better name to distinguish one from another, in the Type column?

Before Samu changed it, they were appimage-type2 and appimage-type3 xD

EDIT: How do we baptize them?

probonopd commented 3 months ago

Well, there are

Type 1 AppImages (those don't use squashfs but ISO9660) (most of them use a dynamic runtime which requires libfuse2, but this is by coincidence, not because they are type 1)
Type 2 AppImages (most of them use a dynamic runtime which requires libfuse2, but this is by coincidence, not because they are type 2)
Type 2 AppImages that happen to use the new runtime from https://github.com/AppImage/type2-runtime (those don't require libfuse anymore because they come with their own, but they do require fusermount; hopefully any version will do once we have implemented this properly)

There may be many different AppImage runtime implementations out there in the wild, not all of which I know. But I am aware of at least some which your table is probably not even taking into account. You should rely only on what the AppImageSpec defines. Everything else is just pure coincidence.

Let me ask: Why have this column at all? Users should not have to care...

ivan-hc commented 3 months ago

Let me ask: Why have this column at all? Users should not have to care...

So a newbie from the more recent Ubuntu and Fedora releases might notice that appimage2 have an issue in running, on the contrary of appimage3

ivan-hc commented 3 months ago

Let me ask: Why have this column at all? Users should not have to care...

So a newbie from the more recent Ubuntu and Fedora releases might notice that appimage2 have an issue in running, on the contrary of appimage3

Also, does not install only Appimages. The database contains 1900 unique AppImages (excluded the 40 scripts for kdegames and 27 for kdeutils), all the rest are portable binaries, scripts or other portable tools.

AppImages have more advantages in AM. For example, the sandbox via Aisap, and other advantages you already know.

ivan-hc commented 3 months ago

Non-Appimages programs have not these advantages.

ivan-hc commented 3 months ago

@probonopd And about "installing an AppImage", in this case, it is worth to use this term.

An installation script:

creates the directory and the remover first (the latter for security reasons)
prepares the AppImage, so...
1. downloads the AppImage
2. puts that AppImage in place
3. extracts the launcher and puts the launcher in the menu
4. same for the icon, that instead is added into a dedicated "icons" directory
creates the AM-updater script, needed to update the AppImage using the points 1 and 2 above, again, if the .zsync file is broken or unavailable

Altogether, all of these steps install files in specific locations, like any distribution package.

The central part you can call "integration" if you want, but overall, it is actually an "installation".

ivan-hc commented 3 months ago

You can see the whole process here https://github.com/ivan-hc/AM/blob/main/templates/AM-SAMPLE-AppImage

ivan-hc commented 3 months ago

kdegames

https://github.com/user-attachments/assets/e7b0f2ea-3d81-4237-92d0-0557a9ce38e3

probonopd commented 3 months ago

And about "installing an AppImage", in this case, it is worth to use this term.

An installation script: (...)

... negates what AppImages are all about:

One app = one file, no installation. Just download, make executable, and run!

ivan-hc commented 3 months ago

... which negates what AppImages are all about:

One app = one file, no installation. Just download, make executable, and run!

Again, its not important the reason something exists, but the way it can be used.

To level a broken chair leg you can use a book.

The developer doesn't care how they use the AppImage: as long as they use it.

And as the developer of the said AppImage, I had a purpose for building it that way. Otherwise I would have built it another way.

If I used a WINE AppImage, double-clicking would open "WineConfig", but it wouldn't launch any game. It would be pointless.

probonopd commented 3 months ago

If I used a WINE AppImage, double-clicking would open "WineConfig", but it wouldn't launch any game. It would be pointless.

The required subset of WINE that a particular AppImage needs should be bundled in that applications's AppImage, as demonstrated in https://github.com/probonopd/libhookexecv/releases/tag/continuous.

It's always the application that should have an AppImage. Not some framework like KDE or WINE. If the application needs parts of such frameworks, the needed parts should be bundled in the AppImage of the application that needs them. Otherwise you create dependencies.

ivan-hc commented 3 months ago

The required subset of WINE that a particular AppImage needs should be bundled in that applications's AppImage, as demonstrated in https://github.com/probonopd/libhookexecv/releases/tag/continuous.

You are talking about "exporting a Windows app into an AppImage"

Doing this for a Windows game may contribute to piracy.

Samueru-sama commented 3 months ago

Alright, sorry for the late reply, I just lost electricty.

Please, if you like the AppImage format, do not call these things "appimage2" and "appimage3". You could write "Note 1" and "Note 2" or something like that.

We use appimage2 and appimage3 to indicate if the appimage doesn't have the libfuse dependency issue, I don't think there is any other short simple way to indicate that in the title.

appimage-dynamic vs appimage-static? that gives the idea that the static appimage can work on any distro like the bundle everything appimages. we are not checking for type1 or type2 appimage, the result will always be appimage2 or appimage3, there will never be a appimage1 or something like that.

And @probonopd if you don't like it because you think it might cause confusion, what do you think of post like this? This is the result of the confusion of the dynamic dependency to libfuse2, which is not made clear by not including in the type spec.

Hopefully in the future when most appimages already use the static runtime, we can drop the 2 and 3 from the name since it won't be needed anymore.

Let me ask: Why have this column at all? Users should not have to care...

I strongly disagree here, I just linked you an example of why they should care.

... negates what AppImages are all about:

One app = one file, no installation. Just download, make executable, and run!

And you can still use them that way, we just provide convenience.

I know there are other integration tools, for example the go-appimage daemon you have, doesn't work with my i3wm, and appimagelauncher has given me problems integrating some appimages, gearlever needs flatpak to be installed which I think is a massive waste of storage for a single application, etc, we also provide update mechanism to all of the appimages.

Also probono if I'm not mistaken you said something on reddit that flatpak and appimage have two different goals? I strongly disagree with that, I think appimages can be expanded to do everything that flatpak already does, we even added a working bubblewrap sandboxing to AM thanks to aisap.

We also try to use a little dependencies as possible, even our install scripts can be run on their own without AM or AppMan.

ivan-hc commented 3 months ago

Personally, I am looking forward to this fusermount issue being finally fixed upstream, so that all AppImages can be used without any problems.

@probonopd I don't like having to classify AppImages into type 1, 2, 3 or X either. But it is the only way, now, to make sure that users understand that, at this time, for what we have, it is impossible to guarantee the functioning of old-generation AppImages on modern systems, out of the box. Especially if the person using them is a new Linux user.

Believe me, it's the only way now to give users time before they judge whether AppImage is a good format or not.

I trust in your abilities.

probonopd commented 3 months ago

Doing this for a Windows game may contribute to piracy.

Not if the author of the Windows game does it. Which, remember, is what the AppImage format is made for - application/game authors wanting to distribute their software to Linux users.

Samueru-sama commented 3 months ago

Not if the author of the Windows game does it. Which, remember, is what the AppImage format is made for - application/game authors wanting to distribute their software to Linux users.

One has to come to the sad reality that not everything will come as an appimage, I think you saw that with MPV, the developers refused over and over again to implement and appimage, they don't even support flatpak, and don't like when distros ship MPV either lmao, to them the only good way to install MPV is using their build scripts, which I used for my MPV appimage.

Very few developers will bother to ship their own game as an appimage, and now that Steam comes as a flatpak it is even less likely because appimages don't work there

I think you also told Ivan that he should request the Steam devs to release an appimage, no way, that's like telling someone to go and ask a dictator to step down lol. Steam is well known for having +10yo issues with lots of activity that haven't been fixed and likely never will, even both the snap and flatpak of Steam are not official do you think we should just give up and not make the appimage for it?

probonopd commented 3 months ago

We use appimage2 and appimage3 to indicate if the appimage doesn't have the libfuse dependency issue, I don't think there is any other short simple way to indicate that in the title.

You could just as well call it *and ** (like footnotes) and then explain what you think needs to be explained about those AppImages. Or See footnote 1 and See footnote 2.

what do you think of post like this?

Probably written by someone who likes IBM Red Hat, Gtk, Gnome, Flatpak, and such. Contains factual misinformation. Doesn't even begin to appreciate the point of AppImages.

Hopefully in the future when most appimages already use the static runtime, we can drop the 2 and 3 from the name since it won't be needed anymore.

Again, static vs. dynamic has nothing to do with the AppImage image type, and there is no type 3 (yet).

And you can still use them that way, we just provide convenience.

Yes, all those things are purely optional.

I know there are other integration tools, for example the go-appimage daemon you have, doesn't work with my i3wm, and appimagelauncher has given me problems integrating some appimages, gearlever

All of these are just workarounds; the proper solution is to get application bundles (like .app, .AppDir, and .AppImage) properly supported in desktop environments.

gearlever needs flatpak to be installed which I think is a massive waste of storage for a single application

Hey, we can agree on that :100:

Also probono if I'm not mistaken you said something on reddit that flatpak and appimage have two different goals?

Yes, completely opposite. AppImage is a self-mounting disk image containing an application bundle, similar to Mac .app or Windows "PortableApps", whereas Flatpak is a distro-agnostic package manager (essentially a "distro on top of the distro").

I think appimages can be expanded to do everything that flatpak already does,

Maybe, but AppImage has different goals. Otherwise we'd have designed something like Flatpak and not something like AppImage.

AppImage is all about: one app = one file, no package managers, no repositories, no dependencies. Just download and run. Everything managed by drag-and-drop and double clicking in the file manager.

probonopd commented 3 months ago

One has to come to the sad reality that not everything will come as an appimage

Life can be cruel. Not everything will come as an .exe or a .dmg.

both the snap and flatpak of Steam are not official do you think we should just give up and not make the appimage for it?

Wait, are you saying you succeeded in making a working Steam AppImage? (Something I had tried a long time ago and given up.) If so, did you let them know how you did it?

Samueru-sama commented 3 months ago

Wait, are you saying you succeeded in making a working Steam AppImage? (Something I had tried a long time ago and given up.) If so, did you let them know how you did it?

Why do you ask? Ivan posted it on reddit and you responded there.

I didn't make it but helped Ivan a bit though.

I did post appimage in one of the issues at steam requesting the appimage though.

However the appimage is basically a mini distro on its own, it ships its own version of mesa even, but that is something I actually wanted to happen, when it comes to gaming you always want the latest version of mesa available.

ivan-hc commented 3 months ago

I didn't make it but helped Ivan a bit though.

it was impossible without your improvements, @Samueru-sama don't be modest, it is 20% mine and 80% yours

probonopd commented 3 months ago

While I admire your work I don't think that feature request in issue #10188 there won't go far because it's already closed. Need to find some other way to explain to them what you did and why it is awesome.

Samueru-sama commented 3 months ago

And btw @probonopd the Steam appimages has its advantages that you can see in the description of the repo

The most important being that it uses its own EAC-Glibc, which archlinux decided to break for some reason.

And the solution before was either, use the Steam flatpak which has its own set of issues, or use an AUR package for Glibc (what could possibly go wrong? lol), or migrate to a gaming distro that still keeps the patch.

ivan-hc commented 3 months ago

While I admire your work I don't think that feature request in issue #10188 there won't go far because it's already closed. Need to find some other way to explain to them what you did and why it is awesome.

Threaten to set us on fire under Valve headquarters? That's an idea!

ivan-hc commented 3 months ago

@probonopd as @Samueru-sama already said

I think you also told Ivan that he should request the Steam devs to release an appimage, no way, that's like telling someone to go and ask a dictator to step down lol. Steam is well known for having +10yo issues with lots of activity that haven't been fixed and likely never will, even both the snap and flatpak of Steam are not official do you think we should just give up and not make the appimage for it?

not to repeat myself always and always

Istantanea_2024-08-08_05-15-59 png

from https://github.com/AppImage/appimage.github.io/pull/3038

ivan-hc commented 3 months ago

It's admirable that you want AppImages to be made "official". But if upstream doesn't want them, we'll have to make do.

probonopd commented 3 months ago

Or use another application.

ivan-hc commented 3 months ago

Or use another application,

I want the AppImage.

ivan-hc commented 3 months ago

To convince upstream developers that something works, there are beta testers.

Third-party packages are all beta testers, unknowingly.

Only if something works, the developer will decide to take it on, officially.

We will have to wait months or years, before being recognized. But in the meantime we must continue to improve our unofficial products.

A package cannot become official overnight because you decided it should be so. It is up to the developer to decide.

Appimage is already difficult to build, for many, then we also have to force the developer to build it, neglecting the effort he has to make in improving the program itself. It's a mess!

Again, one thing is the developer, another is the packager.

It is rare that, in large projects, these two figures coincide.

Take Bottles: 300 contributors, only one packaging format, Flatpak.

probonopd commented 3 months ago

Again, one thing is the developer, another is the packager.

The AppImage format was designed to change this.

But if you want to make unofficial third-party AppImages, then we need at least to think about a way for "mere mortals" (non-technical users who don't know what GitHub is) to be able to know which third-party provided AppImages are high-quality and trustworthy. I think a couple of years ago there were many fake Firefox download sites, all pestered with malware. How to avoid such a situation?

Samueru-sama commented 3 months ago

But if you want to make unofficial third-party AppImages, then we need at least to think about a way for "mere mortals" (non-technical users who don't know what GitHub is) to be able to know which third-party provided AppImages are high-quality and trustworthy. I think a couple of years ago there were many fake Firefox download sites, all pestered with malware. How to avoid such a situation?

We already indicate when an appimage isn't official in am when you do an am -q app

There might be some cases where this slipped thru and one app hasn't been marked as unofficial though.

Also some installation scripts have mutliple options and tell you which is official and which isn't as well, for example Brave:

However I think this can be improved more, I recently talked with Ivan to start flagging the appimages that don't have a transparent build process behind and are just some random binary that's uploaded, that even happens with some official appimages even.

Also notice that AM is more than just for appimages, it can install portable builds of applications, which is very useful for several CLI utilities.

ivan-hc commented 3 months ago

Again, one thing is the developer, another is the packager.

The AppImage format was designed to change this.

and it is still not so.

There are developers who refuse to build their own, and others have had difficulties that led them to abandon it (see Bottles and OBS). We brought you concrete evidence. Even Samu told you that MPV developers want you to compile it yourself! This is the reality!

I am nobody, but an upstream developer will never make excuses like "it costs too much" or "it is difficult" or "we don't have the resources". I have 71 AppImages, how did I build them? Such excuses would never work with me.

They just don't want to do it, they don't want to respond, or they don't want to even try. Or maybe they just want to support another standard, like Flatpak. I think many media outlets agree on this (hence my post on Reddit), support Flatpak, at all costs.

But if you want to make unofficial third-party AppImages, then we need at least to think about a way for "mere mortals" (non-technical users who don't know what GitHub is) to be able to know which third-party provided AppImages are high-quality and trustworthy. I think a couple of years ago there were many fake Firefox download sites, all pestered with malware. How to avoid such a situation?

The answer? You are already into it. In the repository of this issue. Two letters: "AM"

And as @Samueru-sama said before I finishing write this, this can still be improved.

OFF TOPIC: I go sleep, sorry, see you later... and thanks for the attention, @probonopd

ivan-hc commented 3 months ago

Just started working on differentiation between AppImages and other apps, including only unique arguments, all helpers for "kdegames" and "kdeutils" are out from new lists.

I'm running "AM" in Developer Mode, so who wants to check the improvements, will use the "dev" branch of this repo:

am --devmode-enable

or

appman --devmode-enable

I've used the script https://github.com/ivan-hc/AM/blob/dev/programs/appimage-lister-uniq.sh into an offline directory AM-main/programs that you can download using git

Once I made the script executable, it searches exact keywords "appimage" and "mage$", all lowercased, that are available in all installation scripts for AppImages.

These are the new lists for the three architecutures for now implemented in the database:

i686 https://github.com/ivan-hc/AM/blob/dev/programs/i686-appimages
aarch64 https://github.com/ivan-hc/AM/blob/dev/programs/aarch64-appimages
x86_64 (the big one) https://github.com/ivan-hc/AM/blob/dev/programs/x86_64-appimages

Unique x86_64 Appimages now available in the database, excluding helpers for kdeutils (27) and kdegames (40) are 1922.

The first step for the new release will be the filtering of the applications lists.

Almost all non-AppImages programs are official, so we don't need to check if they are official or not.

AppImages instead may ne official or not.

The final step of this process will be the addition of flags "verified" or "unverified", that can be added in "AM" 7.6, eventually.

So to list Appimages in a separated way will be the focus of the release 7.5 of "AM".

ivan-hc / AM

AppImage type detection wrong #818