Support for Ren'py 7.4.11 android releases seems broken

[darkplayground]

I found this project while trying to get android builds to work in the CLI with renpy and I was hoping you had a solution. I followed the instructions and installed everything but when it gets to the "Installing RAPT" section of the renpy install it just sits and hangs for hours and hours. I left it running overnight and it never finished and didn't generate an error. This is the command I was using: renutil install -v=7.4.11 on both ubuntu 20.04 and Debian 11. Both had openjdk8 installed and configured successfully as well as git, wget, libgl1, etc. I'm hoping there's a solution to this because this tool looks to be pretty amazing.

As an aside, I tried using the version of renconstruct from your other repo to do the install and build and got the same results, no error message, just hangs forever at rapt install, even with debug on.

One more thing, in this project you are expected to pass renconstruct a config file but when I used the config format from your other renconstruct repo (man this is getting confusing lol) it generated an error. Looks like this version is expecting TOML and not YAML, though I couldn't find any indication of that in the docs. I used an online convertor to swap the formats and it "worked". Of course it never got to building an android release but it didn't throw that TOML error anymore.

This is what happens btw when I ctrl+c out of the "Installing RAPT" step and just re-run renconstruct:

The command: renconstruct -di /mygame -o /root/artifacts -c /mygame/renconstruct.yaml

[darkplayground]

I found what the error was with installing RAPT. Or at least, I don't think it's because of this script. For some reason openjdk just hangs for some reason every so often on the "building a test application" stage of the installsdk command. I've run it manually after digging through your python and found that it just sits and hangs about 30% of the time no matter what system I'm on. Not sure how to fix that. But at least now I know clearing the cache and re-running renconstruct will make it succeed most of the time.

As for the screenshot I posted, that is still happening even on a successful RAPT install. It seems like the command anatomy has changed between 7.4.8 and 7.4.11. On 7.4.8 the command works fine and does indeed build an android package but on 7.4.11 I get that error about unrecognized command "assembleRelease".

[kobaltcore]

Thanks for reporting this, I'll take a look this afternoon once I'm back from work.

[kobaltcore]

Ah, looks like this has to do with Ren'Py's relatively recent support for universal android application bundles. It generates a new keystore for this and while I accounted for the original one (for normal bundle generation), the new one is not yet auto-generated so it hangs at an interactive point in the installation process. I'll probably push a fix this evening that should resolve this.

[kobaltcore]

Alright, pushed an initial fix for the RAPT installation hangup. I'm going to test-build a few Ren'Py games to see whether the assembleRelease thing is still borked now.

After that I'll also add some extra docs on which config files it expects and the various settings in them.

[kobaltcore]

Okay, the release assembly issue should now be fixed with the latest pushes. Ren'Py 7.4.9 changed how Android packages are being built, so that has now been accounted for. RenKit should now be able to cope with both the old and the new style of CLI invocation.

A build of the tools with these fixes included is available here.

Let me know if this works for you.

[darkplayground]

It seems to be working better and usually getting further but I'm often still stuck on the "installing rapt" stage. I really wish there was some way to see what It was doing during that stage, maybe pass through the output from android.py so I can tell if it's stalled or processing something.

Right now it's failing with an error about a missing keystore command. That seems to be installed with the android sdk but for whatever reason it's not happening?

Extracting
Setting up permissions
Generating Application Keystore
Generating Bundle Keystore
Preparing RAPT
Installing RAPT
tables.nim(234Error: unhandled exception: key not found: keystore [KeyError]
Error: Process completed with exit code 1.

That's the GitHub action stage output. The below config is what I used to run this.

Correct me If I'm wrong but this requires JDK8 right? This is the docker container I'm running from:

FROM --platform=amd64 openjdk:8u312-jdk-slim

ENV DEBIAN_FRONTEND=noninteractive

RUN apt update && apt install -y zip libgl1 git

COPY static/megacmd-Debian_11_amd64.deb ./
RUN apt update && apt install -y ./megacmd-Debian_11_amd64.deb && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY static/renkit-linux.zip /root
RUN unzip /root/renkit-linux.zip -d /usr/local/bin

then my GitHub action looks like this:

jobs:
  build:
    runs-on: self-hosted
    container: darkplayground/renpy:7.4.11
    steps:
      - uses: actions/checkout@v2
        with:
          path: repo
      - name: Build with renkit
        run: |
          renconstruct build -o "$GITHUB_WORKSPACE/dist" -i "$GITHUB_WORKSPACE/repo" -c "$GITHUB_WORKSPACE/repo/renconstruct.tml"

and this is my renconstruct.tml file:

[tasks]
set_extended_memory_limit = true
notarize = false
clean = false
patch = false
overwrite_keystore = false

[patch]
path = "renpy-patches"

[overwrite_keystore]
keystore = "<my b64-encoded keystore>"

[build]
win = false
mac = false
android = true

[renutil]
version = "7.4.11"
registry = "cache"

[notarize]
apple_id = "fill-this-in"
password = "fill-this-in"
identity = "fill-this-in"
bundle = "com.my-organization.my-game"

If it helps feel free to use the docker image darkplayground/renpy:7.4.11, it's public. I installed megacmd on there to eventually upload to mega after the build is done but I haven't been able to get that far with renkit yet.

[darkplayground]

I went ahead and installed the SDK manually again and im still getting the keystore missing command:

root@b5b3fb1d31e2:~/cache/7.4.11/rapt# ../lib/linux-x86_64/python android.py installsdk

I'm compiling a short test program, to see if you have a working JDK
on your system.

The JDK is present and working. Good!

The Android SDK has already been unpacked.

I'm about to download and install the required Android packages. This
might take a while.

[===                                    ] 10% Computing updates...
No updates available
[=======================================] 100% Computing updates...
All SDK package licenses accepted.======] 100% Computing updates...

[=======================================] 100% Unzipping... android-11/framework

I've finished installing the required Android packages.

It looks like you're ready to start packaging games.

root@b5b3fb1d31e2:~/cache/7.4.11/rapt# cd
root@b5b3fb1d31e2:~# renconstruct build -o ~/artifacts -i /game/ -c /game/renconstruct.tml
tables.nim(234Error: unhandled exception: key not found: keystore [KeyError]

[kobaltcore]

This is a little strange because when I run this on a test repo, replicating most of your setup, it works.

My action looks like this:

jobs:
  build:
    runs-on: ubuntu-latest
    container: darkplayground/renpy:7.4.11
    steps:
      - uses: actions/checkout@v2
        with:
          path: repo
      - name: Build with renkit
        run: |
          renconstruct build -i repo -o dist -c repo/config.toml
      - name: Upload Artifacts
        uses: actions/upload-artifact@master
        with:
          name: bundles
          path: 'dist/*'

This executes flawlessly and uploads the built distributions as artifacts. The only thing that I can think of is that you're self-hosting the runner and something may be wonky on it compared to the standardized ubuntu machines on GA.

[darkplayground]

Hmm, ok, maybe something wrong with the game I'm testing? I'm gonna try some changes to that. Fwiw running with the ubuntu-latest resulted in the same error.

Also, something quite annoying is I seem to be getting throttled by the renpy and rapt download server somehow, this is what I get now:

Any ideas why thats happening? When I first was running this program it downloaded at 60+ MBps but now it's so slow, every run takes 10 minutes or more to finish. I've added a GitHub caching step to prevent the download from needing to happen every time but it only caches it if the action succeeds and right now with failure testing that isn't happening.

[darkplayground]

Ok. I'm officially confused. I reverted back to my old build system where I installed and configured renpy manually before I found your renkit and it build the game without issue:

This is the Dockerfile:

FROM --platform=amd64 alpine:latest AS temp
COPY static/renpy-7.4.11-sdk.zip /
RUN unzip renpy-7.4.11-sdk.zip
COPY static/renpy-7.4.11-rapt.zip /
RUN unzip renpy-7.4.11-rapt.zip -d /renpy-7.4.11-sdk/

FROM --platform=amd64 openjdk:8u312-jdk-slim

ENV DEBIAN_FRONTEND=noninteractive

RUN apt update && apt install -y zip libgl1 git

COPY static/megacmd-Debian_11_amd64.deb ./

RUN apt update && apt install -y ./megacmd-Debian_11_amd64.deb && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY --from=temp /renpy-7.4.11-sdk /tmp/renpy-7.4.11-sdk
COPY static/android.py /tmp/renpy-7.4.11-sdk/rapt/
COPY static/interface.py /tmp/renpy-7.4.11-sdk/rapt/buildlib/rapt/
ENV RAPT_NO_TERMS=true
RUN cd /tmp/renpy-7.4.11-sdk/rapt && ../lib/linux-x86_64/python android.py installsdk

as you can see this dockerfile uses your patched android.py and interface.py script to patch out the SSL verification issue and the yesnochoice and input functions of interface but other than that it's vanilla renpy and rapt. I pushed this image up to darkplayground/renpy:temp if you want to give it a try.

This is the GitHub action:

android:
    runs-on: self-hosted
    container: 
      image: darkplayground/renpy:temp
    steps:
      - name: Check out repository code
        uses: actions/checkout@v2
        with:
          path: repo
      - name: "Run linter"
        run: |
          cd /tmp/renpy-7.4.11-sdk
          ./renpy.sh "$GITHUB_WORKSPACE" lint 
      - name: Build with Ren'Py
        id: renpy-build
        run: |
          cd /tmp/renpy-7.4.11-sdk
          ./renpy.sh launcher android_build "$GITHUB_WORKSPACE/repo" --dest "$GITHUB_WORKSPACE/dist"
          echo "::set-output name=build-artifact::$(ls $GITHUB_WORKSPACE/dist)"

I'm so confused as to why it works fine with this renpy but fails with your renkit. Is there a config step or something that your renkit needs to run properly? It looks like that keystore key error is related to something in your nim code. Any ideas?

[darkplayground]

Ok, I dug into the nim code and I've figured out what's wrong. It's back to the config file again. You apparently changed quite a bit with the expected renconstruct config file format and I was still using the only version I found which was from the renconstruct repo. The overwrite_keystore variable needs to be just keystore along with changing the [overwrite_keystore] heading to [keystore] (hence why it was throwing a missing key error) and further down at this line: https://github.com/kobaltcore/renkit/blob/d1a3b7d1a8fab1e7111b3104062e06a6796e0cc2/src/renconstruct.nim#L185 It appears, like most languages, that nim does not like it when you refer to array keys that don't exist. So even though you put the android key check in there for backwards compatibility, it still throws an error when android_apk and android_aab is not found. When I updated my config with those three directives it worked fine.

This is my full proper config now:

[tasks]
set_extended_memory_limit = false
notarize = false
clean = false
patch = false
keystore = false

[patch]
path = "renpy-patches"

[keystore]
keystore = "<my b64-encoded keystore>"

[build]
win = false
mac = false
android_apk = true
android_aab = true

[renutil]
version = "7.4.11"
registry = "cache"

[notarize]
apple_id = "fill-this-in"
password = "fill-this-in"
identity = "fill-this-in"
bundle = "com.my-organization.my-game"

Finally solved. Kinda annoying it was all down to an out of date config file. Hopefully you'll be able to update the docs with this info to prevent anyone else from hitting these issues.

Thanks for all your help.

[darkplayground]

Also, just found this, if I set the android bools to false it fails again saying it can't find the "android" key now lol. So I had to add that key back and set it

[kobaltcore]

Also, something quite annoying is I seem to be getting throttled by the renpy and rapt download server somehow-

Yeah, I've encountered that a couple of times as well, probably put in place to prevent the host from going down, instead just slowing down your download. Ideally Ren'Py would be pre-packaged in your Docker image to prevent any kind of download from there except for once during the build of the image itself.

I recently pushed a Dockerfile to this repository which showcases how this could work, so maybe take a look at that if you want to prevent it from redownloading stuff over and over again.

I'll definitely also be adding some docs to elaborate on the config file and probably also do some extra work to prevent it from failing because of missing keys. Sorry for the trouble.

[darkplayground]

Yeah, I've encountered that a couple of times as well, probably put in place to prevent the host from going down, instead just slowing down your download. Ideally Ren'Py would be pre-packaged in your Docker image to prevent any kind of download from there except for once during the build of the image itself.

Yea, I considered doing that but that reduces the ability for an end-user to switch or use different renpy versions easily, I wanted them to be able to update their renconstruct config with a new version any time and have it download the new release while also keeping the docker image small. The other problem with packaging it is size. The renpy, RAPT, and Android SDK and all that is over 1gb in size which would make the docker image unwieldy. I might provide a pre-cached version for people that want it. But the renkit image I've got now Is a little over 200mb which is really nice lol

I'll definitely also be adding some docs to elaborate on the config file and probably also do some extra work to prevent it from failing because of missing keys. Sorry for the trouble.

No worries man, thank you replying at all. And I'm happy to help you work through these issues, you've done a lot of legwork here that I didn't have to do myself so I'm more than happy to help you improve this tool for us all. Keep up the good work, and please let me know if I can help with anything in the future.