search

triplea-game / triplea

TripleA is a turn based strategy game and board game engine, similar to Axis & Allies or Risk.
https://triplea-game.org/
GNU General Public License v3.0
1.34k stars 393 forks source link

Move install4j bundled JREs from GitHub to Linode #2682

Closed ssoloff closed 6 years ago

ssoloff commented 6 years ago

We bundle JREs with our installer for users that do not have a Java 8 JRE installed on their machine. We currently host these bundled JREs in the triplea-game/assets repo, and the installer downloads them, if needed, directly from GitHub.

There are a few problems hosting these files on GitHub:

A static file server is probably a better place to host these resources rather than using a Git repo. The purpose of this issue is to discuss possibly moving the bundled JREs to one of our Linode servers and serve them from there using Nginx (or something equivalent).

Some issues with self-hosting include:

  1. Increased bandwidth possibly leading to an increased Linode bill. It would be great if we could get some metrics from GitHub to see how often the bundled JREs are downloaded so we can predict how much traffic we'll see.
  2. The host should be available as much as possible. It should not be taken down except for maintenance.
  3. The host name should not change over time. We won't be able to go back and modify installers from older releases to point them to a new host. I'm not familiar with this aspect of our Linode setup, so if changing host names is common, we might have to consider having a reverse proxy or something to redirect from the old name to the new name.

@DanVanAtta @prastle @RoiEXLab @ron-murhammer Thoughts?

DanVanAtta commented 6 years ago

We pay for bandwidth from Linode server, a 350 MB download per build could add up.

Is git LFS an option? I've no personal objection as well to doing history re-writes in assets repo to reduce repo size.

ssoloff commented 6 years ago

a 350 MB download per build could add up

I'll have to double-check, but I don't think the bundled JREs are actually required during the build. Despite the adjective, they aren't actually "bundled" with the installer (the installer only contains their URLs). AFAIK, they are only downloaded when a user running the installer does not have a local JRE installed.

If it turns out they are required during the build, I'll ask forgiveness for opening this issue. :smile: In that case, the history re-write is probably the right solution.

ssoloff commented 6 years ago

Nevermind. Even though install4j doesn't include the JREs with the installer, it still wants to examine them at build time (I guess to verify that they are in its required format). So at least three JREs (Mac, Windows x86, and Windows x64) need to be downloaded per build. :-1:

RoiEXLab commented 6 years ago

@ssoloff From my experience install4j checks if the files exists, and not much more. When we upgraded to install4j7, I temporarily "fixed" the installer generation by simply creating zip files and naming them like the JRE files IIRC. That being said, it seems like a bug that the generator actually requires the files offline.

ssoloff commented 6 years ago

@RoiEXLab Thanks for the suggestion. While an empty file or an empty gzipped tar archive result in the following exceptions:

install4j: compilation failed. Reason: java.io.IOException: The JRE bundle /home/ssoloff/Projects/triplea-game/triplea/build/assets/install4j/windows-x86-1.8.0_144.tar.gz is damaged and cannot be read.
install4j: compilation failed. Reason: JRE bundle does not contain readable tar entries.

A gzipped tar archive with at least one entry apparently makes install4j happy.

So, it looks like we could keep dummy files in the Git repo and the actual files someplace else. In that case, I'm going to re-open this issue as there are some viable alternatives.

ssoloff commented 6 years ago

a 350 MB download per build could add up.

BTW, my eyes weren't working properly when I wrote up this issue. :smile: The bundled JREs are ~35 MiB in size, not ~350 MiB. That order of magnitude probably makes a huge difference.

ssoloff commented 6 years ago

Is git LFS an option?

Possibly. It looks like it will work for devs pretty easily because it integrates with the Git client. However, I'm not sure how one can retrieve the large file from a plain URL, which is how users would actually download the JRE.

I'll set up a test repo and play around with the dev and user workflows.

RoiEXLab commented 6 years ago

@ssoloff Perhaps you should report this as some sort of bug to the install4j guys, because I really don't think it's intended to check the JRe if it's not actually bundling it.

ssoloff commented 6 years ago

@RoiEXLab Good idea. I'll fire off an email to Ingo asking if there is a way around it, or at least what is the reasoning for it.

ssoloff commented 6 years ago

So removing the bundled JREs from the build process is probably not possible. Per Ingo:

I just had a look if I could add a VM parameter to disable the check, but there is some more logic associated with it. The compiler scans the bundle to see if there are any packed files in it, if you provide empty bundles or if the check is disabled, the Windows installer will not work.

I verified the Windows x86 installer I built with a dummy bundled JRE works just fine. However, that simply means the current JRE bundle contains no packed files. I don't think there's any guarantee one or more packed files won't be introduced in the future.

ssoloff commented 6 years ago

I finally got around to trying out Git LFS, and it appears this will work.

I set up a test repo at https://github.com/ssoloff/git-lfs-test, which contains both regular files and files tracked by LFS. I confirmed the bare repository does not actually contain the large files I set up to be tracked by LFS. Manipulating these tracked files in a non-bare repository is seamless, as LFS hides all the gory details from you.

I also confirmed trying to download one of the tracked files through a direct HTTP call to GitHub works as expected. Using wget, I was able to download a tracked file. The only difference I saw from a non-LFS file is that GitHub redirects the request to a different URL (presumably to the server where the large files are actually stored). For example:

$ wget https://github.com/ssoloff/git-lfs-test/raw/master/macosx-amd64-1.8.0_144.tar.gz

is redirected to

https://media.githubusercontent.com/media/ssoloff/git-lfs-test/master/macosx-amd64-1.8.0_144.tar.gz

So the only thing we would have to verify is that the install4j installer follows these redirects when it attempts to download a bundled JRE (I see no reason why it wouldn't).

That just leaves some logistics for the dev team. Git LFS requires an external tool, so everyone would have to install it in order to work with an LFS-enabled repository. I used the RPM package, but they have binaries for all platforms. You can find installation instructions here and play around with your own LFS-enabled repo (or fork my test repo) to see if there are any issues for your platform.

We also need to keep in mind the possibility of installing the LFS client on Travis. Right now, we do a direct HTTP download of the assets we need, but if we ever attempt to clone an LFS-enabled repo, the LFS client must be present. (From travis-ci/travis-ci#3634, it seems that the LFS client may already be pre-installed in the latest Travis images.)

@DanVanAtta @RoiEXLab @ron-murhammer Are there any objections to enabling Git LFS the next time we have to add a large file to the assets repo?

RoiEXLab commented 6 years ago

@ssoloff No objections from my side.

ssoloff commented 6 years ago

@DanVanAtta @ron-murhammer Bumping this comment.

DanVanAtta commented 6 years ago

@DanVanAtta @RoiEXLab @ron-murhammer Are there any objections to enabling Git LFS the next time we have to add a large file to the assets repo?

No objections at all so long as developer workflow instructions are updated on dev docs readme

ssoloff commented 6 years ago

@DanVanAtta Roger that. I'll submit a PR to update the README with the Git LFS client installation instructions.

ssoloff commented 6 years ago

Noting here that, after a brief run with LFS enabled in the triplea repo, it has been disabled because we quickly (~7 days) exhausted our monthly LFS bandwidth quota. See #2921 for more information.