search

yairm210 / Unciv

Open-source Android/Desktop remake of Civ V
Mozilla Public License 2.0
8.26k stars 1.55k forks source link

RFC - Smaller bundled JRE #11091

Closed SomeTroglodyte closed 3 months ago

SomeTroglodyte commented 6 months ago

Before creating

Problem Description

Thinking about those 90MB download zip...

Related Issue Links

No response

Desired Solution

I tried, out of a hunch, to ask the jdk to build a jre adapted to Unciv. The installed size shrinks by ~70MB, the ZIP size by ~9MB.

I asked jdeps which modules are required: jdk\bin\jdeps.exe --print-module-deps --ignore-missing-deps --recursive --multi-release 11 Unciv.jar (supplying an unpacked jar to --class-path and/or --module-path did not change the result). Then I asked jlink to make a runtime: Delete jre from extracted zip, leave exe and jar in place, then: jdk\bin\jlink.exe --add-modules java.base,java.desktop,java.instrument,java.logging,java.management,java.prefs,jdk.unsupported,jdk.crypto.ec --strip-debug --no-man-pages --no-header-files --compress=2 --output .\jre (jdk.crypto.ec was missed by jdeps, missing it blocks any https)

Now jre/lib/modules is 20M instead of 90M, and zipping it back up gives 79.6MiB.

Alternative Approaches

Not investigated whether a similar effect can be achieved using our current packer.

Additional Context

There's not much savings to be had elsewhere - translations is 15M uncompressed, kotlin reflection is >8M of class files, looking into why com.unciv is taking so much space just leads to tons of successively smaller files...

yairm210 commented 6 months ago

Sounds good!!!

SomeTroglodyte commented 6 months ago

I was tending to "not worth the extra hassle". This came from that "can no longer start the gradle 8 builds" issue that doesn't answer our questions, and thus awakened curiosity ("were we not much much leaner not too long ago, in the 10M range???").

What prepares our current jre? Anuken packr? If it has options to control jlink... If not, this would mean one packr run discarding everything packr produces except the exe, then running jlink separately? Also, the module list needs manual maintenance if any major library is added or changed - jdeps autodetection, as shown, fails with a few modules that aren't loaded the normal way. (superficial duckduck skimming suggests the ssl stuff is loaded via reflection) ... also, we're currently pulling the "jre" assets which do not contain these tools, only the "jdk" ones do.

Wait - packr refers to jlink in their readme - will maybe read later.

yairm210 commented 6 months ago

No, LibGDX packr run by cli command

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 15 days.

github-actions[bot] commented 3 months ago

This issue was closed because it has been stalled for 5 days with no activity.

touhidurrr commented 2 months ago

Building JREs from JDK on demand? Cool idea.

touhidurrr commented 2 months ago

I did not even know something like this was even possible. One problem though, how do you plan to build the windows version? Solution can be something like this:

  1. split packr-build task into 3 tasks, build-linux-jre, build-windows-jre, packr-build where packr-build depends on [build-linux-jre, build-windows-jre]
  2. run build-windows-jre on windows-latest obviously.
  3. Instead of using Adoptium API, use actions/setup-java to setup latest temurin JDKs
  4. Now run your commands to build JREs
  5. Use JREs built to pack everything with packr
touhidurrr commented 2 months ago

Found related discussions in: https://adoptium.net/blog/2021/10/jlink-to-produce-own-runtime/

packr support seems sus, the docs are discussed here: https://github.com/libgdx/packr/blob/master/README.md#minimization Their example: https://github.com/libgdx/packr/blob/master/TestAppJreDist/testAppJreDist.gradle.kts

This looks a little too complex to me, jlink solution feels better.

jlink: https://docs.oracle.com/en/java/javase/11/tools/jlink.html

touhidurrr commented 2 months ago

Anyways, I can try to write actions code for this issue with jlink with the approach in https://github.com/yairm210/Unciv/issues/11091#issuecomment-2168830989 if you guys are willing to get onboard.

SomeTroglodyte commented 2 months ago

My brain says: Uuuuugghhhh... There's areas where it feels like a Neanderthal seeing the Monolith - so I agree where I understand, and shy away cowardly where learning is required. I fear to get this automated requires a little trial and error, which I didn't want to back then - I know a little more about actions now, such as how you can get the "release patch" uncivbot to run on your own fork - so actual testing before submitting the script to the boss fork is possible... But add some laziness, and I'm happy if someone else does all that.

touhidurrr commented 2 months ago

ok, I tried the tutorial here: https://adoptium.net/blog/2021/10/jlink-to-produce-own-runtime/

Here is the result:

312M    jdk-11.0.23+9      // unpacked official temurin 11 jdk
125M    jdk-11.0.23+9-jre  // unpacked official temurin 11 jre
187M    jdk.tar.gz         // official temurin 11 jdk
42M     jre.tar.gz         // official temurin 11 jre
53M     minimized-jre      // the minimized jre that i generated
53M     Unciv.jar          // latest Unciv jar

So, 125M to 53M is like ~57.6% save. And yes, I tried running Unciv, it works.

touhidurrr commented 2 months ago

Ok, there one problem though. Maybe this is not as easy as it looks. I tried running Unciv with minimized-jre and network calls seems to be not working. Works fine with jdk shipped jre.

~$ ~/test$ ./minimized-jre/bin/java -jar Unciv.jar
sh: 1: xdg-mime: not found  // ignore this, common across all jres
2024-06-15T10:42:25.721715Z [threadpool-daemon-1] [Github] [ERROR] Exception during GitHub download | javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:209)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:208)
        at com.unciv.logic.github.Github.download(Github.kt:52)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic(Github.kt:208)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic$default(Github.kt:200)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invokeSuspend(ModManagementScreen.kt:288)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.utils.ConcurrencyKt$launchCrashHandling$1.invokeSuspend(Concurrency.kt:89)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandling$1.invoke(CrashHandlingExtensions.kt:17)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher.dispatch$lambda$0(Concurrency.kt:190)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

2024-06-15T10:42:25.945385Z [threadpool-daemon-1] [Github] [ERROR] Exception during GitHub download | javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:209)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:208)
        at com.unciv.logic.github.Github.download(Github.kt:52)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic(Github.kt:208)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic$default(Github.kt:200)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invokeSuspend(ModManagementScreen.kt:288)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.utils.ConcurrencyKt$launchCrashHandling$1.invokeSuspend(Concurrency.kt:89)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandling$1.invoke(CrashHandlingExtensions.kt:17)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher.dispatch$lambda$0(Concurrency.kt:190)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

2024-06-15T10:42:25.946604Z [threadpool-daemon-1] [ModManagementScreen$tryDownloadPage] [ERROR] Could not download mod list | java.lang.NullPointerException
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invokeSuspend(ModManagementScreen.kt:288)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.utils.ConcurrencyKt$launchCrashHandling$1.invokeSuspend(Concurrency.kt:89)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandling$1.invoke(CrashHandlingExtensions.kt:17)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher.dispatch$lambda$0(Concurrency.kt:190)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
touhidurrr commented 2 months ago

Anyways, here are the steps that I used so that you guys can try debugging.

wget https://api.adoptium.net/v3/binary/latest/11/ga/linux/x64/jdk/hotspot/normal/eclipse -O jdk.tar.gz
tar -xzvf jdk.tar.gz
rm jdk.tar.gz
mv jdk-11* jdk-11
./jdk-11/bin/jdeps -s Unciv.jar > deps.txt
UNCIV_MODULES=$(cat deps.txt | grep -v 'not found' | cut -d ' ' -f 3 | tr '\n' ',' | sed 's/,$//')
./jdk-11/bin/jlink --add-modules $UNCIV_MODULES --strip-debug --no-man-pages --no-header-files --compress=2 --output minimized-jre
wget https://github.com/yairm210/Unciv/releases/download/4.11.19-patch1/Unciv.jar
./minimized-jre/bin/java -jar Unciv.jar
touhidurrr commented 2 months ago

I think this will take time to figure out. The best for now maybe just to merge #11751 for now as we try to figure this out. I don't think can fix this dependency issue alone or this will get done by the next version or anytime soon. We need to fix this issue in such a way so that we don't need to touch it in future. As for now, this is not working as intended:

./jdk-11/bin/jdeps -s Unciv.jar > deps.txt
UNCIV_MODULES=$(cat deps.txt | grep -v 'not found' | cut -d ' ' -f 3 | tr '\n' ',' | sed 's/,$//')
SomeTroglodyte commented 2 months ago

network calls seems to be not working

This one I did cover up there... Look for "(jdk.crypto.ec was missed by jdeps, missing it blocks any https)" in the first post. A search turns up some explanations hinting why, and hinting this is pretty common IIRC.

touhidurrr commented 2 months ago

This one I did cover up there... Look for "(jdk.crypto.ec was missed by jdeps, missing it blocks any https)" in the first post. A search turns up some explanations hinting why, and hinting this is pretty common IIRC.

So, just adding jdk.crypto.ec manually will solve this issue?

SomeTroglodyte commented 2 months ago

Yes, but 4 eyes see more than 2, so don't buy that bag untested, or don't let me prejudice you on which Unciv features to thoroughly test for missing modules from jlink minification. 'Later, I need to do some RL.

touhidurrr commented 2 months ago

don't buy that bag untested

That is the concern I have also. How to know if the JRE we have shipped will work? We can do trial and error by actually shipping it first then debugging but that would be hilarious (maybe something like putting the cart before the horse also). About workarounds for this concern, I can't think of any.

touhidurrr commented 2 months ago

Ok found a solution that looks like it might resolve any such unforseen dependency missing issues. It is here: https://stackoverflow.com/a/62359298 But we need the maven commands gradle counterpart.