search

AppImageCommunity / libappimage

Implements functionality for dealing with AppImage files
https://appimage.org
Other
46 stars 29 forks source link

Please remove or make explicit the dependency on libgobject, libcairo, librsvg #104

Closed probonopd closed 5 years ago

probonopd commented 5 years ago

Getting lots of ERROR: appimage_register_in_system : Unable to load libgobject-2.0.so.

Is this dependency coming from libappimage? I don't remember having seen it in pre-libappimage days.

appimaged, continuous build (commit 2e34378), build 108 built on 2019-03-14 17:43:20 UTC
probonopd commented 5 years ago

Interestingly, this is contained in

sudo apt-get install libglib2.0-dev

Why on earth would we need a -dev package to run appimaged?

probonopd commented 5 years ago

Interestingly, a self-built libappimage does not show a dependency on libgobject-2.0.so:

me@host:~/libappimage/build$ ldd ./src/libappimage/libappimage.so
    linux-vdso.so.1 (0x00007ffeb23a7000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f14cc07b000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f14cbe77000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f14cbc5a000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f14cb8cc000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f14cb6b4000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f14cb2c3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f14cc618000)

However, inspecting it with strings does make it a likely candidate for having introduced this dependency "secretly":

me@host:~/libappimage/build$ strings ./src/libappimage/libappimage.so | grep goblibgobject-2.0.so
libgobject-2.0.so.0
libgobject-2.0.so.0.0
probonopd commented 5 years ago

In fact, there are even more "hidden dependencies":

me@host:~/libappimage/build$ strings ./src/libappimage/libappimage.so | grep -e  "^lib.*so.*"
libpthread.so.0
libdl.so.2
libz.so.1
libstdc++.so.6
libgcc_s.so.1
libc.so.6
libappimage.so.1.0
librsvg-2.so.2 # What about this? We cannot assume it to be everywhere
libcairo.so.2 # What about this? We cannot assume it to be everywhere
libgobject-2.0.so # What about this? We cannot assume it to be everywhere
libgobject-2.0.so.0 # What about this? We cannot assume it to be everywhere

The same is true for appimaged which uses libappimage internally:

me@host:~$ strings squashfs-root/usr/bin/appimaged | grep -e  "^lib.*so.*"
(...)
libstdc++.so.6
libgcc_s.so.1
(...)
librsvg-2.so.2 # What about this? We cannot assume it to be everywhere
libcairo.so.2 # What about this? We cannot assume it to be everywhere
libgobject-2.0.so # What about this? We cannot assume it to be everywhere
probonopd commented 5 years ago

In fact, it's even way worse than that. If we make the "hidden dependencies" to real dependencies, and then check with ldd, we see that libappimage and hence appimaged now depend on a whole lot of stuff indirectly:

# "Unhide" the "hidden dependencies" 
me@host:~/libappimage/build$ /isodevice/Applications/patchelf --add-needed librsvg-2.so.2 --add-needed libcairo.so.2 --add-needed libgobject-2.0.so ./src/libappimage/libappimage.so

# Check the real dependencies
me@host:~/libappimage/build$ ldd ./src/libappimage/libappimage.so
    linux-vdso.so.1 (0x00007ffe98b28000)
    libcairo.so.2 => /usr/lib/x86_64-linux-gnu/libcairo.so.2 (0x00007f348923c000)
    libgobject-2.0.so => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so (0x00007f3488fe8000)
    librsvg-2.so.2 => /usr/lib/x86_64-linux-gnu/librsvg-2.so.2 (0x00007f3488db0000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3488b91000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f348898d000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3488770000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f34883e2000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f34881ca000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3487dd9000)
    libpixman-1.so.0 => /usr/lib/x86_64-linux-gnu/libpixman-1.so.0 (0x00007f3487b34000)
    libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f34878ef000)
    libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f348763b000)
    libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f3487409000)
    libxcb-shm.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007f3487206000)
    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f3486fde000)
    libxcb-render.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007f3486dd1000)
    libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f3486bc7000)
    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f348688e000)
    libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f348667c000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3486474000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f34860d6000)
    libglib-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f3485dc0000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f3485bb8000)
    libgdk_pixbuf-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007f3485994000)
    libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f34855f6000)
    libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007f34853e9000)
    libpangoft2-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0 (0x00007f34851d3000)
    libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007f3484f86000)
    libcroco-0.6.so.3 => /usr/lib/x86_64-linux-gnu/libcroco-0.6.so.3 (0x00007f3484d4b000)
    libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f348498a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3489998000)
    libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f3484758000)
    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f3484554000)
    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f348434e000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f34840dc000)
    libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f3483ed8000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f3483cb0000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f3483a95000)
    libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007f3483841000)
    libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007f34835a3000)
    libthai.so.0 => /usr/lib/x86_64-linux-gnu/libthai.so.0 (0x00007f348339a000)
    libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f3482fe3000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f3482dbd000)
    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f3482ba8000)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f348295b000)
    libgraphite2.so.3 => /usr/lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007f348272e000)
    libdatrie.so.1 => /usr/lib/x86_64-linux-gnu/libdatrie.so.1 (0x00007f3482527000)
    libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f348097e000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f3480777000)
probonopd commented 5 years ago

At the very least, the dependencies must be made explicit (so that they show up in ldd) so that tools like linuxdeploy and linuxdeployqt will have a chance to bundle the dependencies (e.g., here). We will have to see how large appimaged becomes then.

probonopd commented 5 years ago

https://artifacts.assassinate-you.net/artifactory/appimaged/travis-112/appimaged-x86_64.AppImage has 2.9 MB as opposed to 0.7 MB before. Do we think this is still acceptable? What functionality do we gain that was previously unavailable?

probonopd commented 5 years ago

Bundling this stuff is painful: https://github.com/AppImage/appimaged/pull/78. g_type_check_instance_is_fundamentally_a is caused by bundling usr/lib/libgobject-2.0.so needs yet another workaroud and more stuff to be bundled.

Now we are at 3.8 MB.

So, can we link libgio-2.0.a statically to get around this?

azubieta commented 5 years ago

https://github.com/AppImage/libappimage/blob/f3d6400ce1c33b96e8671a971ebcb0df4a63faed/src/libappimage/utils/IconHandle.cpp#L238

https://github.com/AppImage/libappimage/blob/master/src/libappimage/utils/IconHandle.cpp

This is where those libraries are used. As you could imagine they are for creating thumbnails.

probonopd commented 5 years ago

Can you make them properly linked instead of dlopen()ed?

dlopen() should only be used for optional features (like libnotify) where we can degrade gracefully in case the library is not there.

As you could imagine they are for creating thumbnails.

Can you elaborate a bit? Which functionality do those libs do that cannot be achieved by other means?

azubieta commented 5 years ago

https://github.com/AppImage/libappimage/blob/f3d6400ce1c33b96e8671a971ebcb0df4a63faed/src/libappimage/utils/IconHandle.h#L11-L18

Basically all the image manipulation done in libappimage.

probonopd commented 5 years ago

Do we really need to manipulate images other than moving and copying them around? E.g., are we using those libraries to write the metadata required by the XDG Thumbnailer standard? Could we use other tools that can be statically linked to do the same?

azubieta commented 5 years ago

Do we really need to manipulate images other than moving and copying them around?

The thumbnails spec says:

image format for thumbnails is the PNG format, regardless in which format the original file was saved

So yes if we want to do thumbnails generation right we must do image manipulation. By example to transform svg icons into png.

probonopd commented 5 years ago

Makes sense, but I think we could move the image conversion to AppImage generation time (=appimagetool), and ensure that .DirIcon is compliant to the thumbnails spec?

We could possibly even embed the needed metadata with some dummy padding which we could then edit using fast string manipulation at integration time.

Example: appimagetool writes .DirIcon as a thumbnailer-compliant PNG with e.g., Thumb::URI=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(...) which we could then replace when the AppImage is being integrated.

What do you think? Of course we would need to update the AppImageSpec accordingly.

azubieta commented 5 years ago

I think we could move the image conversion to AppImage generation time

That could be done and will fix the issue for the newly created AppImage but what about the old ones?

This could help performance but the produced files we will be larger.

azubieta commented 5 years ago

Actually I like the idea of doing this at build time. And yes we should make a room in the AppDir for thumbnails. The AppIcon may not be always a good candidate i.e. when it's an svg file.

TheAssassin commented 5 years ago

It's .DirIcon, and that's what's specified by ROX Filer. All you can do is put a proper icon there, but I wouldn't bother investing too much work into this. Furthermore, I am pretty sure the thumbnail standards don't have "multiple files" features or so. And that's not needed either if you ask me.

You can IMO render an SVG if needed to there at build time, for instance.

probonopd commented 5 years ago

The AppIcon may not be always a good candidate i.e. when it's an svg file.

Do you mean the .DirIcon? Then let's define that people need to use a thumbnailer-suitable one.

what about the old ones?

Do whatever appimaged did in pre-libappimage days (=nothing?).

azubieta commented 5 years ago

We will remove all the image manipulation feature and the libs loaded using dl.

.DirIcon will be used as thumbnail if it's a png file. Otherwise I purpose to use a placeholder icon.

appimagetool should receive this feature.

:+1: if you agreed

probonopd commented 5 years ago

I agree, once

In the meantime, we can live with the status quo (now that we have a workaround for appimaged).

probonopd commented 5 years ago

What about the Thumb::URI=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(...) stuff? Do we need to mandate those placeholders too?

TheAssassin commented 5 years ago

Part of the standard, so yes. We noticed quite late. I think that's also supposed to make things work on KDE, isn't it?

probonopd commented 5 years ago

Yes.

probonopd commented 5 years ago

Then we need to define what exact placeholders with what exact length we require in the .DirIcon.

azubieta commented 5 years ago

There is a PR on the AppImageSpec to make the .DirIcon follow the Thumbnails Format Specification.

appimagetool issue created https://github.com/AppImage/AppImageKit/issues/958

The current implementation is actually better than the older (which actually does nothing). And uses the 'do nothing' feature as fallback. So until the appimagetool receive such feature and a while after have no need to touch the code.

probonopd commented 5 years ago

As as side effect when we use Thumb::URI=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(...) and friends, we may bring back Thumbnail copying into runtime.c because it is going to be much simpler to implement properly then (using only copying and string manipulation).

azubieta commented 5 years ago

I've been working on this, as we thought building and linking libcairo and and librsvg is possible. Also it's possible to do the same with libgobject but linking this library statically is the mother of all chaos. glib is required by a large amount of apps therefore they will be linking to it if in a single application you use to different versions of glib (one statically linked and other dynamically) a crash is certain. So, this is a NO-GO.

@probonopd would you please provide more information about the system where you are getting those errors. So I could try improving the dl opening process.

A far as I can see the solution to this is to move all image processing code to appimagetool and leave to libappimage just the copy file and set thumbnail metadata features.

TheAssassin commented 5 years ago

There is no reason to link libgobject, it's not used directly AFAIK. It's just a dependency of rsvg.

azubieta commented 5 years ago

The source of the error is libgobject not being found.

ERROR: appimage_register_in_system : Unable to load libgobject-2.0.so.

Even if we link librsvg statically libgobject will be missing.

probonopd commented 5 years ago

What needs it? Can't we get rid of what needs it?

TheAssassin commented 5 years ago

There is no reason to link librsvg statically.

probonopd commented 5 years ago

Err, why not? Linking it and its dependencies statically, I mean.

azubieta commented 5 years ago

librsvg links to libgobject. Get rid of libgobject means losing thumbnails generation of svg .DirIcon

@probonopd

glib is required by a large amount of apps therefore they will be linking to it if in a single application if you use to different versions of glib (one statically linked and other dynamically) a crash is certain. So, this is a NO-GO.

Update: libcairo also depends on libgobject therefore we will not be able to perform any image manipulation.

TheAssassin commented 5 years ago

Err, why not? Linking it and its dependencies statically, I mean.

It must be linked statically? Please tell me why. Seriously, it's not needed to link it statically. We can ship the .so files.

azubieta commented 5 years ago

Thumbnails generation and proper icons deployment (icons being deployed on the right folder according to their size) are features that didn't exists before libappimage-1.0.0. Therefore my proposal is to be more silent on this kind of errors and use the old libappimage behavior (just copy the file) as fallback.

As far as I can remember we do just copy the file if something goes wrong. But it may need a review. What do you think?

TheAssassin commented 5 years ago

Not sure if we need a fallback, it's just more complexity... but I'd be okay with that, as long as I don't have to maintain that... There should be a warning though.

I think we should just dynamically link this and call it a day.

probonopd commented 5 years ago

Dynamic linking is ok if we can make an AppImage bundle that does not run into issues like this one: https://github.com/AppImage/appimaged/issues/82

https://github.com/RazrFalcon/resvg is not an option? They write

librsvg is heavily tied to GNOME, which makes it painful to distribute outside the Linux ecosystem

;-)

TheAssassin commented 5 years ago

Seems like a valid suggestion, the library looks good at a first glance. I'm pretty sure we had a look at it already, though, and there was a reason not to use it... But I don't remember. @azubieta please check whether switching to that library is an option.

azubieta commented 5 years ago

resvg also requires pango. Optionally it could use Qt5 as back-end but it's the same situation see: https://github.com/RazrFalcon/resvg/blob/master/BUILD.adoc#cairo-backend

probonopd commented 5 years ago

Isn't pango about fonts which we don't need? Could it be disabled?

TheAssassin commented 5 years ago

Only if we build it ourselves. And that doesn't eliminate the need for the GNOME libs, there's still gobject deps in cairo.

probonopd commented 5 years ago

In any case, then we need to make a proper appimaged AppImage that bundles all dependencies in a way that makes the whole thing work; as a proof-point that build products using libappimage can be AppImage'ified properly.

TheAssassin commented 5 years ago

Not a fan of the rendering quality, but I've checked ImageMagick (libmagick6) and it doesn't seem to depend on RSVG or other GNOME dependencies. Its SVG extra codec rather has a direct dependency on libxml2, so I guess they implement their own SVG parser. That's the only real alternative to RSVG I know that can render vector graphics properly to raster graphics.

> lddtree /usr/lib/x86_64-linux-gnu/ImageMagick-6.9.7/modules-Q16/coders/svg.so
svg.so => /usr/lib/x86_64-linux-gnu/ImageMagick-6.9.7/modules-Q16/coders/svg.so (interpreter => none)
    libMagickCore-6.Q16.so.3 => /usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.so.3
        liblcms2.so.2 => /usr/lib/x86_64-linux-gnu/liblcms2.so.2
        liblqr-1.so.0 => /usr/lib/x86_64-linux-gnu/liblqr-1.so.0
            libglib-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
                libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3
        libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3
        libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
            libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1
        libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6
            libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6
            libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1
                libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6
                libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6
                    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0
                        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
            libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
                ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1
        libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2
        libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60
            libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60
            libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6

Not sure how trivial it is to ship that library, though. It's a fair bit more complex than librsvg/libcairo, since it has this dynamic codec support thing. But, on the other hand, we don't have to build it, I guess; we can just ship it as it is.

It's definitely worth investigating shipping of libmagick, I guess with a plugin for linuxdeploy we could get proper results. I'd love to have RSVG as an optional dependency, though, since its SVG rendering quality is outstanding, I use it all the time through rsvg-convert. I'd implement it as a "let's link to libmagick, then try to load RSVG dynamically; if it's there, let's use it, otherwise just use ImageMagick" or so.

azubieta commented 5 years ago

We found at https://github.com/AppImage/appimaged/issues/82 that we can safely use libcairo and librsvg. @probonopd is proposing linking the libs directly to libappimage and not using dlopen. I guess that it's safe path to go and will allow us to remove a lot of fragile 'dlopen' related code.

@TheAssassin are you okay with it ?

TheAssassin commented 5 years ago

If it works reliably, yes, sure. I'd like to keep the dlopen code for now (as in, keep it in the codebase, even if it won't be used directly any more), though. I posted on IRC an idea how to allow for using different backends, that way everyone can build libappimage against the libraries that work best for them. Through dynamic loading, though, we can implement a mechanism to use the "best available backend" for the job then, taking the bundled one (whatever that is) as fallback. We can discuss that separately, though. Let's fix this first.

azubieta commented 5 years ago

issue fixed by #118 Will close it, please reopen if required.