search

ungoogled-software / ungoogled-chromium

Google Chromium, sans integration with Google
BSD 3-Clause "New" or "Revised" License
20.38k stars 829 forks source link

Add Flatpak, AppImage, or Snap support #36

Closed nyancat18 closed 6 years ago

nyancat18 commented 8 years ago

Could you add a flatpak o appimage format
http://flatpak.org/ http://appimage.org/

probonopd commented 8 years ago

I am already converting Chromium snapshots from https://download-chromium.appspot.com/dl/Linux_x64?type=snapshots to AppImages in a fully automated way using Travis CI. It would be easy to do for ungoogled-chromium if there are deb packages, too.

AppImage has the advantage that it runs on most (also older) distributions.

nyancat18 commented 8 years ago

@probonopd

https://github.com/Eloston/ungoogled-chromium/releases/tag/51.0.2704.106-1

probonopd commented 8 years ago

Can you provide ones for debian oldstable or trusty? That way the resulting AppImage could run on more but the very newest distros.

Eloston commented 8 years ago

@probonopd Can Chromium even build on those platforms? Do your Chromium AppImages run on those systems? Are there enough people that use distributions that old?

This is a good idea, but it will take quite a bit of work. The Debian and Ubuntu package building process is based off of Debian's build of Chromium, so it has more system dependencies than Google's build of Chromium. This includes building with the system's Ninja builder, clang compiler, and linking against the system's FFmpeg libraries (which are all sensitive to the versions, except maybe Ninja).

If we are to get this work by modifying the Debian build process, there are several things that will have to be done. The meta-build configuration flags and patches to work with system libraries will have to be removed/disabled, the buildtype might need to be switched to Official, and some tools or scripts that have broken as a result of domain substitution or source cleaning will have to be provided or fixed in some way. If we are going to get this to work on Debian Wheezy or Ubuntu Trusty, some things might break beause the tools on those systems are pretty old.

Unfortunately, I don't have enough time to execute what I've just stated. I am planning on upgrading these patches to the latest version of Chromium, and I have priorities outside of this project that are taking up my time. However, I will have some time to provide help and implement a process into buildlib. So if anyone wants to take up the task, go right ahead.

probonopd commented 8 years ago

"Are there enough people that use distributions that old?" - I cannot answer that but I do know that projects like Firefox and LibreOffice put a lot of care to build on old systems to maximize compatibility.

Eloston commented 8 years ago

@probonopd Well those seem to be the earliest versions that Chrome supports, so I guess it can be done somehow.

Eloston commented 8 years ago

41 made me realize that my approach to this problem was wrong. I was too narrowly focused on modifying the Debian build process to realize the approach described in #41 would be better.

Once #41 is done, adding support for AppImage should be relatively easy.

SandNerd commented 7 years ago

Since AppImage can handle updating the package as well, this approach might be suited to be the "official" way to provide the app. In addition to less work building for different distros, this also minimizes the bugs related to dependencies hell (e.g. #107). I'm suggesting this especially since you're not able to support the project as much as you used to.

I personally use Ubuntu so having this application on APT (#37) would be nice. Imho though, focusing more on unGoogling chromium should be the highest priority and where the majority of the fine work be. If someone sees the convenience of installation/updating as a higher priority, they can always install chromium/chrome.

Eloston commented 7 years ago

@sahal2080 You have some solid reasoning. That may not be a bad idea.

probonopd commented 7 years ago

Can you get it to build on Travis CI?

Eloston commented 7 years ago

@probonopd I haven't tried it yet, but I've had this concern in the past.

ghost commented 7 years ago

Would be great to see flatpak or appimage replace all the existing builds. Now that the stand-alone build is possible this should be easy I think. I always thought flatpak had better integration features than appimage but not sure. FP has sandboxing which I don't think AI has (by default?). Sandboxing would probably be a good idea for an additional layer of security.

probonopd commented 7 years ago

Providing an AppImage would have, among others, these advantages:

Here is an overview of projects that are already distributing upstream-provided, official AppImages.

FP has sandboxing built in but FP needs to be installed on the target system first.

ghost commented 7 years ago

Works for most Linux distributions - so does Flatpak One app = one file = super simple for users - so is Flatpak No unpacking or installation necessary - if by "installing" you mean have launcher icons created for the easy launching of games and apps then this is a feature that users want so both formats need to create those. Also, if you mean AI keeps running from the same downloaded file and doesn't make a copy into a hidden location, that's annoying because the user would lose the app if they cleared out their downloaded files. I'd rather it was "installed" so I could save the installation package if I so chose to do so but that had no impact on me continuing to run the app. No root needed - same for flatpak, plus sandboxing No system libraries changed - same with flatpak. Works out of the box, no installation of runtimes needed - runtimes, if they are used, are automatically downloaded and installed. Having shared libraries is a good thing and means FP packages are going to be smaller than AI packages since AI has no library sharing. Optional desktop integration - FP's integration is default AFAIK Optional binary delta updates - FP's default GPG2-signing - FP has it Works on Live ISOs - FP can be installed on liveISOs just like any program can be, and if FP was included by default if it takes off and becomes a good standard, then obviously installation wouldn't be needed. Can use the same AppImages when dual-booting multiple distributions - I'd think you wouldn't need to duplicate the app data with either system. Can be used with different sandboxes of the users' choice (e.g., Firejail, Bubblewrap) - I'm not sure if FP can work with alternate sandboxing, but as long as FP uses the best one and there's no big reason to use a different one, that wouldn't matter much.

probonopd commented 7 years ago

Sorry @Swiftpaw but some of your claims are simply not correct (e.g., FP needs to be installed first, scatters files into the filesystem, and does not work on Live systems, just to name 3). In order not to further spam this thread, let's close this here though.

ghost commented 7 years ago

Uh what? Did you even read (and comprehend) anything I wrote? Or do you try to shut down anyone whom you think is biased even though I was simply listing the features of each package system that I know about? You were the one who sounded biased by listing several features that you made sound exclusive to AI over FP which is totally incorrect as my original reply pointed out.

"FP needs to be installed first"...while AI doesn't, is how you were intending that? This wasn't a bullet point mentioned, but that's correct if that's what you meant. If you're referring to "works for most Linux distros", both FP and AI are compatible with most Linux distributions, but FP isn't available out-of-the-box, not until it comes standard. It's easy to install on all those distros, though. So yeah, AI is a little better in this regard, so what's your problem here? Explain yourself.

"Scatters files into the filesystem", I was arguing that FP probably does this while AI might not? Given that FP needs to be installed first, and that FP "installs programs", I assume AI doesn't need to copy files elsewhere while FP does. My point was that this isn't necessarily a bad thing. I like installing packages and not having to keep the package file around on my desktop if I want to run the same app again in the future. So, what are you trying to say about this part?

"Does not work on Live systems", the only thing you mentioned was "FP", so I assume you're talking about FP here? Regardless, I was saying that FP does work in Linux environments, and that both will do so. You can run anything in a live environment because it's just a RAM disk. There is nothing particularly special about putting program data into a RAM disk instead of a HDD. You can even download, install, and run games in a live environment as long as that live environment enabled sufficient graphics drivers to do so. So, explain yourself here again too, what about this do you disagree with? Since that's the defensive attitude you're showing.

Both FP and AI have some features that the other doesn't have, but overall they are fairly comparable and they are both step ups above the old static binary method and either solution would be better than trying to make repos for every distro out there. I'm in favor of both, but you, probonopd, acted like the things you listed are exclusive features of AI over FP, and that's definitely wrong, hence my original response.

Eloston commented 7 years ago

@Swiftpaw @probonopd Hey guys, please stay to the original topic of adding Flatpak or AppImage support. (Adding support for both will be fine too).

Eloston commented 7 years ago

I will accept a pull request to add Flatpak or AppImage support (or both) into utilikit. Chances are you'll need a configuration type like linux_conservative and new build types for the build files generator.

Eloston commented 6 years ago

Things have changed for version 64. One can use the linux_portable base bundle to build, and then add new packaging types for Flatpak and AppImage.

The existing packaging types are pretty simple and should work as a good reference for these ones. If anyone is interested in this, feel free to submit a PR.

rugk commented 6 years ago

Flatpak support would be great, as it runs this here in a sandbox then.

probonopd commented 6 years ago

How does one use the linux_portable base bundle to build?

Eloston commented 6 years ago

@probonopd I have implemented a new packaging system (in the redesign branch) that is simpler and more straightforward. In the next few days, I will be working to merge it into master and then begin updating to version 68. If you are willing to wait, it should be easier to work with the new system.

Eloston commented 6 years ago

@probonopd 68 is out now. You can have a look at the tar building instructions as a reference for your packaging scripts. The design doc's packaging section also changed a bit to reflect the new design.

Let me know if you need clarifications.

probonopd commented 6 years ago

Can you please point me at what I have to do on Ubuntu 14.04 (the oldest still-supported LTS release) to get the dependencies in place?

$ sudo apt-get -y install clang-6.0 lld-6.0 llvm-6.0-dev python python3 ninja-build
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package clang-6.0
E: Couldn't find any package by glob 'clang-6.0'
E: Couldn't find any package by regex 'clang-6.0'
E: Unable to locate package lld-6.0
E: Couldn't find any package by glob 'lld-6.0'
E: Couldn't find any package by regex 'lld-6.0'
E: Unable to locate package llvm-6.0-dev
E: Couldn't find any package by glob 'llvm-6.0-dev'
E: Couldn't find any package by regex 'llvm-6.0-dev'

Thank you.

Eloston commented 6 years ago

@probonopd I don't believe it is necessary with linux_portable config bundle to build against the oldest Ubuntu version in order to maintain compatibility.

That being said, LLVM provides APT repos all the way back to Trusty. The package names for those tools seem to be the same. I'll add this into the instructions for convenience.

probonopd commented 6 years ago

Running into

$ ./get_package.py linux_simple build/src/ungoogled_packaging
Traceback (most recent call last):
  File "./get_package.py", line 18, in <module>
    from buildkit.common import (ENCODING, BuildkitAbort, get_logger, validate_and_get_ini,
  File "/home/travis/build/probonopd/ungoogled-chromium/buildkit/common.py", line 42
    return schema_dictcast({configparser.DEFAULTSECT: object, **data})
                                                               ^
SyntaxError: invalid syntax

Reference: https://travis-ci.org/probonopd/ungoogled-chromium/builds/420681072#L599-L606

probonopd commented 6 years ago

Using Python 3.6 (the newest one I can use) seems to remove that error, but now I am running into

+./tools/gn/bootstrap/bootstrap.py -o out/Default/gn -s
  File "./tools/gn/bootstrap/bootstrap.py", line 80
    print 'Building gn manually in a temporary directory for bootstrapping...'
                                                                             ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(t 'Building gn manually in a temporary directory for bootstrapping...')?

Reference: https://travis-ci.org/probonopd/ungoogled-chromium/builds/420686866#L1305

How can I enforce ./tools/gn/bootstrap/bootstrap.py -o out/Default/gn -s to use Python 2.7?

probonopd commented 6 years ago

With python2 I now get

+python2 ./tools/gn/bootstrap/bootstrap.py -o out/Default/gn -s
  File "./tools/gn/bootstrap/bootstrap.py", line 115
    exec(response, _globals, _locals)
SyntaxError: unqualified exec is not allowed in function 'CallPythonScopeScript' it is a nested function
Eloston commented 6 years ago

@probonopd

Using Python 3.6 (the newest one I can use) seems to remove that error

I developed the code on 3.5 (the only Python 3 available in stretch), so 3.5 is the minimum. Using 3.6 shouldn't hurt, though.

SyntaxError: unqualified exec is not allowed in function 'CallPythonScopeScript' it is a nested function

I have seen this error once before on an older CentOS 7 machine. At the time, I didn't have much time to investigate.

After reading around a bit, there may be some differences in how exec is implemented in different Python 2 versions.

From the latest Python 2 docs (2.7.15 at the time of writing), it is supposed to work:

The first expression may also be a tuple of length 2 or 3. In this case, the optional parts must be omitted. The form exec(expr, globals) is equivalent to exec expr in globals, while the form exec(expr, globals, locals) is equivalent to exec expr in globals, locals. The tuple form of exec provides compatibility with Python 3, where exec is a function rather than a statement.

But, I keep seeing reports of people explicitly rewriting the exec function-like syntax into the old statement form. It might be originating from this bug, which was fixed in 2.7.9: https://bugs.python.org/issue21591

Debian stretch ships with 2.7.13. What version of Python 2 are you using?

probonopd commented 6 years ago

After having pulled in Python 3.6.3 from a PPA, this error seems to be gone and I now have a new error:

/home/travis/build/probonopd/ungoogled-chromium/build/src/base/template_util.h:145:46: error: no template named 'is_trivially_copy_constructible' in namespace 'std'
using is_trivially_copy_constructible = std::is_trivially_copy_constructible<T>;
                                        ~~~~~^
1 error generated.
[7/367] CC base/third_party/libevent/evrpc.o
ninja: build stopped: subcommand failed.
Command '['ninja', '-C', '/tmp/tmp3nKiQf', 'gn']' returned non-zero exit status 1

https://travis-ci.org/probonopd/ungoogled-chromium/jobs/420804430#L2399-L2405

Eloston commented 6 years ago

@probonopd Should be fixed in 674540188d48d997ea4bb02d0f19621743f64b29

probonopd commented 6 years ago

Thank you. More errors... https://travis-ci.com/probonopd/ungoogled-chromium/builds/82908706#L2463-L2488

Eloston commented 6 years ago

@probonopd In that case, you may want to do what Arch Linux does for Python 2 on these lines:

Out of curiosity, what platform are you building on that has python as Python 3? I thought you were building on Trusty?

probonopd commented 6 years ago

Yes, I am building on Trusty. Here is my .travis.yml.

probonopd commented 6 years ago

In that case, you may want to do what Arch Linux does for Python 2 on these lines

@Eloston I don't know where to insert those lines. Since you know a lot about how to build Chromium (and I don't), do you think you could clone my .travis.yml and get it to work up to the line https://github.com/probonopd/ungoogled-chromium/blob/67483c0f3ee6fd5817f5af90a1b54b67b1fc1b8e/.travis.yml#L43? Then I could take over from thereon.

Thanks for your help!

Eloston commented 6 years ago

@probonopd The Python issue may be because you specified "3.6" on top. Maybe you could specify "2.7" and then install 3.6 after the fact via apt so that python becomes Python 2?

However, you seem to be trying to fit an elephant in the room by building the source code in Travis CI. It is something we investigated in #37 and it definitely won't work (Chromium's build process is too computationally demanding to be willingly built for free by most service providers). So you'll either need to pull from ungoogled-chromium-binaries for new Portable Linux binaries, or delegate compilation to a willing third-party service.

probonopd commented 6 years ago

The latest binaries on https://github.com/ungoogled-software/ungoogled-chromium-binaries/releases are from Dec 19, 2016?

EDIT: There is more at https://ungoogled-software.github.io/ungoogled-chromium-binaries/ (why not on GitHub Releases?). Are they working on trusty?

probonopd commented 6 years ago

so that python becomes Python 2

I am doing this explicitly here:

https://github.com/probonopd/ungoogled-chromium/blob/67483c0f3ee6fd5817f5af90a1b54b67b1fc1b8e/.travis.yml#L35

probonopd commented 6 years ago

Can you describe how e.g., https://ungoogled-software.github.io/ungoogled-chromium-binaries/releases/linux_portable/64bit/67.0.3396.87-2 gets created? Where is it compiled, where are the build logs, etc.? It would be most likely really trivial to hook in AppImage generation into the existing pipeline in this case.

Eloston commented 6 years ago

Where is it compiled, where are the build logs, etc.? It would be most likely really trivial to hook in AppImage generation into the existing pipeline in this case.

I like your approach of learning backwards (i.e. from the implementation back to the design, like I do for Chromium), but I have design docs so you don't need to do this. Have a look at the design doc section for packaging to get an idea of this part of the process. Also, have a look at the README for ungoogled-chromium-binaries to see what it's all about.

From those, I think we can figure out a workflow that suits us both.

probonopd commented 6 years ago

Not sure I read the 2nd document correctly, but are you saying there is no CI build pipeline (Jenkins, GitLab CI, Travis,...) and project members/you are building on local machines? If so, what kind of local machines (which distribution)?

Eloston commented 6 years ago

are you saying there is no CI build pipeline (Jenkins, GitLab CI, Travis,...) and project members/you are building on local machines?

That is correct. For some background, I have declined offers in the past for dedicated building hardware because I did not want to handle that kind of responsibility. I also did not want others to feel obligated to provide the hardware either. It was a questionable perspective in hindsight, but that's where we are at now.

If so, what kind of local machines (which distribution)?

The latest versions of Portable Linux were built on the same machine I use to build Debian stretch packages. I don't know what compatibility is like on very old or new distributions, but no one has reported any issues of that nature.

probonopd commented 6 years ago

That is correct. For some background, I have declined offers in the past for dedicated building hardware because I did not want to handle that kind of responsibility. I also did not want others to feel obligated to provide the hardware either. It was a questionable perspective in hindsight, but that's where we are at now.

I fully understand your decision (and have made my fair share of mistakes trusting random people lending housed hardware in the past).

So, essentially if I write a bash script that takes the Portable Linux build and turns this into an AppImage, that's all you'd need?

Eloston commented 6 years ago

So, essentially if I write a bash script that takes the Portable Linux build and turns this into an AppImage, that's all you'd need?

Yep. I'm thinking that we can add a script like package_appimage.sh into linux_simple that runs after build.sh to generate the AppImage (and rename package.sh to package_tar.sh to prevent confusion). This way, we can generate multiple different packages directly from the build outputs.

FYI I'm not sure if you will need this, but the buildkit command filescfg list (which uses filescfg_generator() in buildkit.filescfg) will list out only the files needed to run Chromium.

probonopd commented 6 years ago

I have written an initial recipe to convert the existing build into an AppImage:

https://github.com/AppImage/AppImages/blob/master/recipes/ungoogled-chromium.yml

Feedback appreciated.

Eloston commented 6 years ago

@probonopd Two potential issues:

  1. You have written quite the shell command to determine the Portable Linux binary to download, but it's assuming that I will be the uploader. This will fail if someone else uploads a Portable Linux binary.
  2. There is a slight branding change of the browser. I'm guessing this is unavoidable because of the nature of AppImages (e.g. a user has a real Chromium AppImage side-by-side with an ungoogled-chromium AppImage). If this is not the case, then I'd prefer if the change were reverted.

Everything else LGTM.

probonopd commented 6 years ago

Thanks for your feedback @Eloston.

  1. How can I find the URL to the latest version? Do you have a permalink URL that redirects to the latest, for example? We could also change it so that instead of downloading something it copies in some local deb, but for that we would at least know the name and location of that deb.
  2. Are you referring to the fact that I change the desktop file to say "Ungoogled Chromium" instead of "Chromium"? I think this is the correct thing to do since the name of the application is actually "Ungoogled Chromium", and we need to somehow distinguish it as something different from upstream Chromium. Please keep in mind that users may have multiple versions of Chromium on their machine, both Ungoogled and upstream. What should I do?
Eloston commented 6 years ago

@probonopd

How can I find the URL to the latest version? Do you have a permalink URL that redirects to the latest, for example? We could also change it so that instead of downloading something it copies in some local deb, but for that we would at least know the name and location of that deb.

Unfortunately, there's no way to create a permalink using only GitHub Pages. I haven't really thought of using one before, so I never looked into it. Do you happen to know of any free permalink services that are simple enough to be updated via CI?

The other alternative would be to parse the INI file, but that won't be trivial with just shell commands (it's already hard enough to get the version to use).

we need to somehow distinguish it as something different from upstream Chromium. Please keep in mind that users may have multiple versions of Chromium on their machine, both Ungoogled and upstream

I see. Thanks for confirming my suspicions in the previous message.

In that case, we can just leave it alone. But for aesthetics, could you change it to "ungoogled-chromium" or "Chromium (ungoogled)"? Seeing it as "Ungoogled Chromium" looks a little weird to me.

probonopd commented 6 years ago

You have written quite the shell command to determine the Portable Linux binary to download, but it's assuming that I will be the uploader. This will fail if someone else uploads a Portable Linux binary.

Who else could be the uploader? And where would those files be hosted? Can the name of the uploader be determined dynamically? How?

The other alternative would be to parse the INI file

Like so?

export VERSION=$(wget -q "https://raw.githubusercontent.com/Eloston/ungoogled-chromium/master/version.ini" -O - | grep chromium_version | cut -d " " -f 3)-$(wget -q "https://raw.githubusercontent.com/Eloston/ungoogled-chromium/master/version.ini" -O - | grep release_revision | cut -d " " -f 3)
echo $VERSION

This currently gives me https://github.com/Eloston/ungoogled-chromium-binaries/releases/download/68.0.3440.106-1/ungoogled-chromium_68.0.3440.106-1_linux.tar.xz but that file does not exist, there are only debs. So looking at version.ini doesn't look like a viable strategy to me.

I have not quite understood what the problem is in the current method of getting the most recent available *_linux.tar.xz. Can you please elaborate and describe the optimal strategy? Thanks.

Eloston commented 6 years ago

@probonopd

Who else could be the uploader? And where would those files be hosted? Can the name of the uploader be determined dynamically? How? I have not quite understood what the problem is in the current method of getting the most recent available *_linux.tar.xz. Can you please elaborate and describe the optimal strategy? Thanks.