search

OtterBrowser / otter-browser

Otter Browser aims to recreate the best aspects of the classic Opera (12.x) UI using Qt5
https://otter-browser.org
GNU General Public License v3.0
1.8k stars 277 forks source link

Investigate Appimage #1148

Closed pierreporte closed 7 years ago

pierreporte commented 8 years ago

For a really long time, installation of software under Linux was possible by using distribution-specific package manager or by compiling the whole programme. The package managers work really well but if the desired programme is not present or is too old, the compilation was the only way and it may be really long and not easy for someone not really into computers. For several months now, two thing came out to simplify the installation of software under Linux: Flatpak (formerly known as XDG-Apps) for all distributions and Snap for Ubuntu only. Flatpak and Snap packages are distributed like Windows installers and provide sandboxed applications. For now, according to the Flatpak website, it is available in Archlinux, Debian, Fedora, Mageia and Ubuntu.

I think it would be good to provide Flatpak packages for Otter. Flatpak being available to Ubuntu, Snap could be avoided. Users could install each weekly or beta version without having to compile or wait that the package maintainers do it. With Flatpak only, virtually all distributions providing Flatpak can be targeted and it could even be official packages.

The process of building a Flatpak package seems quite easy and fast so the additional work should be minimal after the first time.

Personally, even if I compile Otter directly from the git repository (from the AUR package in Archlinux), I will use the Flatpak package if it is made available.

Emdek commented 8 years ago

@pierreporte, well, since it will take distributions some time to adopt QtWebKit NG it could make sense to prepare such packages to not let users wait.

annulen commented 8 years ago

If all you need is standalone binaries + GUI installer, flatpack is insane overkill. I propose to use more lightweight solutions like QtIFW (see CPackIFW), AppImage, or something in this vein

pierreporte commented 8 years ago

@annulen I don’t get the “overkillness” of Flatpak by reading the documentation of both Flatpak and AppImage. Could you do a fast little summary of the pro and cons of Flatpak, Snappy, Appimage and QtIFW?

Besides this, we also need to consider which packaging system is the most used. It would be a pity that the user would have to install a whole packaging system for one program.

annulen commented 8 years ago

There is no need to install anything to use Qt IFW. You download installer and execute it. That's all. I was using IntallJammer in the past, there was no Qt IFW those days. But it's probably dead by now.

I don’t get the “overkillness” of Flatpak

It runs application in a chroot with almost complete OS, which is distrributed within the package, while right approach is to pack just Qt and a few other libraries, and use libc, X11 and all other stuff fronm the host system.

probonopd commented 8 years ago

right approach is to pack just Qt and a few other libraries, and use libc, X11 and all other stuff fronm the host system

Which you can do with AppImage @annulen

pierreporte commented 8 years ago

Can AppImage packages be integrated to the system like any other application? That is, Otter will automatically appear in the right menus and could be set as the default browser? Also, if someone installs Otter with the package manager provided by the distribution and with the AppImage package, what happens?

probonopd commented 8 years ago

An AppImage is just a self-mounting ISO that contains an application and everything it needs to run which cannot be assumed to be part of the base system. Hence, the application inside the AppImage can copy a .desktop file into the system to register itself with the system (e.g., to show itself in the menus). There is a desktopintegration helper script which can be used for this purpose.

If someone installs Otter with the package manager provided by the distribution and with the AppImage package, then both are available and can be used.

See appimage.org for information; a large number of example AppImages (including some browsers) are available for testing on https://bintray.com/probono/AppImages.

pierreporte commented 8 years ago

It’s great then. We could then have classic packages for stable releases and AppImage packages for weekly and beta without any interference.

probonopd commented 8 years ago

Yes, for updating e.g., from weekly to weekly, binary delta updates could be used (so that users only download the bytes that have changed, making updates quick and efficient).

pierreporte commented 8 years ago

According to Jesse Smith’s post Opinion on DistroWatch, AppImage clearly seems to be the better packaging system.

ahoneybun commented 8 years ago

I managed to get a snap working in --devmode, it does not have flash working at the moment. Working on those issues now.

pierreporte commented 8 years ago

I was willing to propose a Chocolatery package for Windows, since Firefox and Chrome are already available, and it has very similar features to the package managers on Linux (installation, un-installation, update, dependencies). However, someone already did one but it is clearly outdated. In fact, it is made by @adgellida who clearly states that his Chocolatery packages are unmaintained. Should we consider to ask him to give us control on the package?

Creating Chocolatey Packages

mariokamenjak commented 8 years ago

Doesn't Otter have auto update on Windows?

Emdek commented 8 years ago

@mariokamenjak, we have support for auto update, but currently it is used only as update notification. I'm considering moving Windows installer to Qt IFW, since it would allow to split main package into modules and do only partial updates (usually only main exe gets changed anyway), using 7Z for archives, plus would allow to install other modules and could be used as update tool. I guess that it would be best to create semi offline installer that bundles common libs, QtWebKit backend and selected translations, allowing to fetch more locales and other modules (like QtWebEngine backend or development files) during installation.

annulen commented 8 years ago

In this case it makes sense to provide Qt IFW packages for all platforms

mariokamenjak commented 8 years ago

Well, I am glad that a snap package is in the pipeline though.

probonopd commented 8 years ago

Is there something I could help you with regarding the AppImage?

Emdek commented 8 years ago

@probonopd, I'm going to give it a try for this beta (which means this week, in worst case after beta itself), this way Linux users could make use of revived QtWebKit which is here even bigger update than under Windows. ;-)

pierreporte commented 8 years ago

Great ! I recently tried the AppImage file for OpenShot 2.1 (the new stable version after 3 years of work!) and it didn’t work. Can’t wait to test the Otter package.

Emdek commented 8 years ago

While beta got delayed after all I'm still planning to try to create Appimage for one of upcoming weeklies.

probonopd commented 8 years ago

Generated an experimental AppImage based on the trusty ppa, please try: https://bintray.com/probono/AppImages/Otter/_latestVersion#files I did not test extensively, so it is expected that some fine-tuning might still be needed.

Here is the recipe that was used to produce it: https://github.com/probonopd/AppImages/blob/master/recipes/otter/Recipe

pierreporte commented 8 years ago

@probonopd I tried your AppImage and it doesn’t work. It asks me if I want do create a .desktop file, but right after it quits saying that libwrap.so.0 is missing.

/home/thomas/.cache/thumbnails/normal/6c7214fca28c029ee7a412cce876ceea.png is missing. Probably not running ./bin//otter-browser.wrapper from within an AppImage.
Hence falling back to using .DirIcon
/tmp/.mount_O8mH5U/usr/bin/otter-browser: error while loading shared libraries: libwrap.so.0: cannot open shared object file: No such file or directory

I use an up to date Archlinux where Otter runs usually fine.

probonopd commented 8 years ago

Indeed libwrap0 did not end up inside the AppImage although it is supposed to. This was a bug in the recipe. Have changed it and triggered a new build; should be up in a few minutes.

annulen commented 8 years ago

Note that making network request through libwrap is going to harm performance of browser.

probonopd commented 8 years ago

@annulen for my proof-of-concept I used the existing debs for trusty; seemingly they have a dependency on libwrap. Another route would be to compile specifically for the AppImage, and to minimize external dependencies für this build.

probonopd commented 8 years ago

Fixed version is up; works for me on Ubuntu 16.04, please try on as many different distros as possible.

pierreporte commented 8 years ago

It works now but there are some bugs with the language. I did not test extensively but in the preference window, the buttons to apply the new parameters are in English and not in the system language. Of course, everything is right in the classical Otter.

probonopd commented 8 years ago

What might cause this? Is it not picking up the translations?

pierreporte commented 8 years ago

It is for nearly everything, but not for dialog box buttons (OK, Apply, Cancel). I found other bugs, but I have to reproduce them to determine precisely where it is—I’ll post them later.

By the way, running the AppImage gives the following warnings before doing anything else. It way help you to refine your package.

/home/thomas/.cache/thumbnails/normal/6c7214fca28c029ee7a412cce876ceea.png is missing. Probably not running ./bin//otter-browser.wrapper from within an AppImage.
Hence falling back to using .DirIcon
/tmp/.mount_7PFLBr/usr/bin/otter-browser: ../lib/x86_64-linux-gnu/libgpg-error.so.0: no version information available (required by /usr/lib/libgcrypt.so.20)

(otter-browser:1547): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='common'

(otter-browser:1547): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='latin'
Warning: QSslSocket: cannot resolve SSLv3_client_method (ssl/qsslsocket_openssl_symbols.cpp:125, void {anonymous}::qsslSocketCannotResolveSymbolWarning(const char*))
Warning: QSslSocket: cannot resolve SSLv3_server_method (ssl/qsslsocket_openssl_symbols.cpp:125, void {anonymous}::qsslSocketCannotResolveSymbolWarning(const char*))
Emdek commented 8 years ago

@probonopd, use of daily PPA is bad idea and defaulting to Trusty is even worse. :-P We don't produce packages for that outdated version anymore since it is ancient and incompatible.

Any suggestions which distribution could be best as base system? Probably we should simply use newest Qt from SDK with custom compilation of revived QtWebKit.

BTW, should typical Appimage contain both 32 and 64 bit versions? It might end up quite big.

probonopd commented 8 years ago

@Emdek my AppImage is just a proof-of-concept and I am hoping that this project will provide official builds. You are entirely free to decide what to put into an AppImage, since the AppImage is really just a fancy self-mounting ISO file.

There are two main ways this can be done:

  1. Compile specifically for the AppImage; e.g., if you are building on Travis CI, you can add a script that bundles up the build artefacts as an AppImage
  2. Use pre-existing deb packages (this is what I used)

In any case, you should make sure to build on a not-too-recent system, because otherwise your AppImage will depend on recent base system dependencies (e.g., glibc) which older distributions will not have available. So if you want to create an AppImage that users of older distributions can use, then you should compile on, e.g., debian oldstable, Ubuntu trusty or older, or CentOS 6 or something like that. Note that the latest Qt is available for these systems. You can use Qt from the Qt company or, e.g., from https://launchpad.net/~beineri/+archive/ubuntu/opt-qt57-trusty. If you don't care about users of older distributions, then you can disregard that.

AppImage lends itself especially well to the distribution of continuous/nightly builds, since you can have more than one in parallel and they don't screw up your base system. So I'd suggest that you provide both stable/release and nightly/continuous/testing builds.

Each AppImage is for one architecture only. The MuseScore project, for example, provides AppImage builds for x86_64, i686, and armhf (Raspberry Pi).

Emdek commented 8 years ago

@probonopd, sure, but it would be still better to use something that is not at least half year outdated in examples. ;-) Then I guess that I'll go for Qt from SDK (they know what are doing and it works great on various distributions, just once I had to create some symlink or install some other version of curl or something else) and try to get similar build environment for revived QtWebKit (perhaps directly from their CI). So we will need two separate packages for 32 and 64 bit, that is fine, at least they will be smaller than one containing both... I'm hoping to find some time to prepare it soon, perhaps it could be based on build machine by @Frenzie, producing both DEB packages (we can keep them, they are small and we need to build binaries anyway) but against custom Qt installation and invoking Appimage script.

probonopd commented 8 years ago

I'll remove my example once you have yours up, just ping me when that's the case.

pierreporte commented 8 years ago

I was wondering, how AppImage would handle the documentation (see #1181)? It works for thing like icons but what about a bunch of HTML files meant to be opened by the browser?

annulen commented 8 years ago

Why should it handle documentation? It's a bunch of static file added to archive, or even qrc resource compiled into binary

pierreporte commented 8 years ago

It shouldn’t necessarily handle the documentation, but in the eventuality of an acceptance of my documentation system and an integration of it in the AppImage binary, I want to know if any problem may arise. For example, what about the displayed URLs and the link behaviour.

mytskine commented 7 years ago

I wanted to help this wonderful project, especially now that glibc2.23 (Debian testing) breaks Opera12. So I tried to build an AppImage. In order to automate the process, I used Vagrant (over VirtualBox). This was intended as a proof of concept, with everything in a single file, but I couldn't make it work. I publish it here so that someone more knowledgeable can finish it, or just get inspiration.

I don't know if Flatpak or Snap provide any tool to help packaging, but I found that packaging a complex application for AppImage is a real pain. The first hard part is to recursively find the dependencies and put them in the right place: recursive ldd wasn't enough, I had to use strace, and then understand how Qt loads its binary plugins and in which directories it looks for them. Yet, this isn't enough: some of these dependencies have to be removed so that the host libraries are used instead. I haven't found any documentation on this, yet a few examples show that the goal is to remove a few core libraries from the package until nothing conflicts with the host libraries. I don't know if a solution always exists.

For instance, when Otter is built on a Debian stable|oldstable basis, the AppImage with every libraries will crash on my Debian testing system. If I remove the old core libc library, the ABI breaks with some unknown symbols. If I remove each other libc files that has an ABI conflict, then I have a segmentation fault. I have no idea how to get out of this.

probonopd commented 7 years ago

You might want to have a look at linuxdeployqt. Be warned, work-in-progress, still has some bugs but hey, it addresses exactly the gripes with bundling Qt apps you describe.

In the meantime, why not repackage pre-existing debs as an AppImage? That is the easy route for a project that already has debs. That's how I do these. Some fine-tuning and testing on many distributions might required, and maybe adding and/or removing a bundled library or two.

This is the recipe used: https://github.com/probonopd/AppImages/blob/master/recipes/meta/Otter.yml

ahoneybun commented 7 years ago

I did get it in a snap but flash does not work, I think its an upstream issue or that I'm missing a package for the snap.

On Tue, Oct 4, 2016, 4:15 PM probonopd notifications@github.com wrote:

You might want to have a look at linuxdeployqt https://github.com/probonopd/linuxdeployqt. Be warned, work-in-progress, still has some bugs but hey, it addresses exactly the gripes with bundling Qt apps you describe.

In the meantime, why not repackage pre-existing debs as an AppImage? That is the easy route for a project that already has debs.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OtterBrowser/otter-browser/issues/1148#issuecomment-251499868, or mute the thread https://github.com/notifications/unsubscribe-auth/AEqJ0qJl9uxnlkeh1PofBs5Zz7y3UAHPks5qwrPbgaJpZM4Izi4K .

Aaron Honeycutt

mytskine commented 7 years ago

Two things I forgot to mention in the previous comment:

probonopd commented 7 years ago

Can you find out what tries to load libgcrypt.so.20 from the target system? Nothing inside the AppImage does that...

probonopd commented 7 years ago

@mytskine

especially now that glibc2.23 (Debian testing) breaks Opera12.

Oh no, another glibc breakage? Can you provide some details?

annulen commented 7 years ago

@mytskine

IIRC, as of Qt 5.7 the old web engine is deprecated.

That's not exactly true, QtWebKit was removed from official Qt SDK binaries but source code is released for each version since then. We are working on revived QtWebKit, see http://qtwebkit.blogspot.com/2016/08/qtwebkit-im-back.html for more details. Plan is to make new release alongside with Qt 5.8

mytskine commented 7 years ago

@probonopd

  1. I'll try to use linuxdeployqt. I'm reluctant to use the current DEB packaging, because it uses the libraries packaged by Debian, which means that QWebEngine is not part of it. If I understand correctly, this reduces the features of Otter. Again, I may be wrong, but this QWebEngine/QtWebKit may be the reason why the browser from my locally compiled beta11 (QWebEngine) seems much more responsive than the deb of beta10 (QtWebKit).
  2. Since I upgraded to libc 2.23 (and now 2.24) I randomly a get fatal error when I use Opera. Some pages are more "dangerous" than others, but the same page may crash the browser 3 times straight, yet Opera will load it flawlessly on the fourth attempt. The error message is always Unexpected error 9 on netlink descriptor XX.
  3. How can I know what loads gcrypt? If I just look inside the AppImage, I have the impression that the otter-browser binary requests it.
% fuseiso Otter-0.9.99.glibc2.14-x86_64.AppImage tmp && cd tmp/
% LD_LIBRARY_PATH="./usr/lib/x86_64-linux-gnu;./usr/x86_64-linux-gnu" ldd usr/bin/otter-browser | fgrep gcrypt
    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20
probonopd commented 7 years ago

@mytskine

I'll try to use linuxdeployqt

That's great but please keep in mind that it is still very new and you might run into rough edges; especially Qml is not yet handled correctly. Be sure to have a look on its issues page, especially #2.

Since I upgraded to libc 2.23 (and now 2.24) I randomly a get fatal error when I use Opera.

Thanks for the hint, something to watch out for.

How can I know what loads gcrypt?

Inside the AppDir, run grep -r libgcrypt.so.20 . and it should give you the files that try to load it.

mytskine commented 7 years ago

@probonopd

Your suggestion of grep -r libgcrypt.so.20 . displays nothing. I discovered that ldd -v displays the details of its recursive resolution of dynamic linking, so with the afore mentionned LD_LIBRARY_PATH trick, I could track it down :

The system libgcrypt.so.20 is loaded by libsystemd.so.0 which is loaded by libdbus-1.so.3 which is required by several libraries inside the AppImage (libcgmanager, libpulse, libQt5Multimedia, etc).

Emdek commented 7 years ago

@probonopd, these DEBs are built against legacy QtWebkit, and we would prefer to use appimage as a way to let users use the revived version.

About gcrypt, it's not used directly, it was listed as optional dependency in the past but it was recently removed.

annulen commented 7 years ago

@Emdek You can build DEBs (or RPMs) containing anything you want, e.g. same contents as you put into appimage. If fact, I believe there is a number of users that would prefer this way to appimage.

Emdek commented 7 years ago

@annulen, sure, but it requires at least twice the total packages size, and publishing releases already takes a lot of time, thanks to my relatively slow upload speed.

Frenzie commented 7 years ago

@annulen Sure, us Debuntu users would prefer a DEB and Fedora users would prefer an RPM, etc. That's exactly why AppImage is a better solution at least in principle. ;-) Plus it has AppImageUpdate, although for the moment that's somewhat cumbersome to use.

Incidentally, I came across some documentation here https://github.com/probonopd/AppImageKit/wiki/Bundling-Qt-apps