Performance issues on Android

[griffi-gh]

(Android F-droid 1.3.524) l i changed settings from maximum to minimum and back.
now it periodically freezes for 5 seconds unless I set graphics to minimum.
it's unplayable. newer version runs slower overall too

[divVerent]

This is odd, I cannot reproduce that yet; what was the last version on F-Droid that didn't have this issue yet?

[griffi-gh]

idk, i haven't played for a while, just installed all updates and it's really slow

[divVerent]

Main relevant recent changes:

v1.3.511 to v1.3.524: one change to input latency (had some stuff in wrong order), sure, doesn't change this; otherwise only visual changes.

v1.3.469 to v1.3.511: newer Ebitengine, some refactorings that should have been equivalent, some different approaches for loading, and anti-aliasing on the map in the menu.

HOWEVER, on F-Droid there's another relevant change: v1.3.511 was released twice, once as all-platforms APK and once as four separate APKs, one per platform. Any chance you might have gotten the wrong one (e.g. arm instead of arm64)?

[divVerent]

Last version without split APKs: https://f-droid.org/repo/io.github.divverent.aaaaxy_103305110.apk

Also useful to test:

• https://f-droid.org/repo/io.github.divverent.aaaaxy_103305114.apk (same but arm64 only)
• https://f-droid.org/repo/io.github.divverent.aaaaxy_103030469.apk
• https://f-droid.org/repo/io.github.divverent.aaaaxy_103030457.apk
• https://f-droid.org/repo/io.github.divverent.aaaaxy_103030436.apk

To be fair, I do not know how to downgrade without losing savegame and settings other than by using adb:

$ adb pull /sdcard/Android/data/io.github.divVerent.aaaaxy/files/save/save-0.json aaaaxy-save-0.json
$ adb shell pm uninstall io.divverent.github.aaaaxy
$ adb install io.github.divverent.aaaaxy_103305110.apk
<now launch the game, touch the leap of faith checkpoint, and quit it right away, so it creates the necessary directory>
$ adb push aaaaxy-save-0.json /sdcard/Android/data/io.github.divVerent.aaaaxy/files/save/save-0.json
<now launching the game will start from the new savegame>

So I'd recommend doing that to find out if any of the currently published versions do not have the problem yet.

Savegames are compatible across all 1.3.* versions.

[griffi-gh]

it sometimes resolves itself after teleporting.
Medium mode runs fine at full framerate, but high and above are unplayable due to freezes. I'm running the arm64 version

[divVerent]

The only difference between Medium and High is that High adds the fading away of not visible anymore areas instead of them disappearing right away.

This however means the game has to remember the previous frame, which is an extra frame copy operation, combined of course with a more complex render pass as the final render is now a three way blend.

Nothing really changed in the game regarding that render pass though - it might be a change of the underlying Ebitengine library that abstracts graphics usage here. To locate it, it would help to find the last version without the issue.

On Sat, May 13, 2023, 19:58 griffi-gh @.***> wrote:

it sometimes resolves itself after teleporting. Medium mode runs fine at full framerate, but hihg is unplayable

— Reply to this email directly, view it on GitHub https://github.com/divVerent/aaaaxy/issues/223#issuecomment-1546771270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB5NMDZPXQ7LSRRLINA6EDXGAN2HANCNFSM6AAAAAAYAVFBBU . You are receiving this because you commented.Message ID: @.***>

[griffi-gh]

1.3.43 runs perfectly fine... (oldest 1.3.x version available)

(btw downgrading can be done by just adding -r -d to pm install, unless the app forbids it)

[griffi-gh]

Here are the versions i tried running:

• 1.3.524 - affected
• 1.3.43 - ok
• 1.3.303 - affected
• 1.3.103 - ok
• 1.3.219 - ok
• 1.3.263 - ok

[griffi-gh]

ok, so something happened between 1.3.263 and 1.3.303

also 1.3.303 crashed my system ui after a while because i left it running in background...
seems like it also has a memory leak?

[divVerent]

v1.3.303 is the first one that uses Ebitengine 2.5.

I do not think I can revert that, and on my other devices, performance is comparable or even better.

As for crash - any chance you can get adb logcat messages from around the crash? The game can't actually run in background - it stops all processing if switched away from. However, it does take up a lot of RAM.

Which device are you on, with how much RAM?

[griffi-gh]

samsung A52s (android 13, oneui 5.1), with 6gb of ram,

by crash I mean the entire os slowed down to a crawl before it kicked me back to the lockscreen (it happened after i went back to my homescreen without closing the game and then opened firefox).
usually stuff like this is caused by excessive ram usage

[griffi-gh]

btw, maybe the freezes are gc pauses?

[divVerent]

I somewhat doubt that, as then it would not differ so much across quality levels - but good idea to add config options for Go's GC behavior so it can be experimented with on Android and iOS too. Elsewhere one can use environment variables.

Will add that soon.

On Sun, May 14, 2023 at 1:20 PM griffi-gh @.***> wrote:

btw, maybe the freezes are gc pauses?

— Reply to this email directly, view it on GitHub https://github.com/divVerent/aaaaxy/issues/223#issuecomment-1546951979, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB5NMHOUPNEWXO7FBHFWVLXGEH4DANCNFSM6AAAAAAYAVFBBU . You are receiving this because you commented.Message ID: @.***>

[divVerent]

In next release, you can edit config.json to include:

  "debug_go_gc_percent": 400,

to see if that really yields 4 times fewer, but in exchange longer, GC pauses. That should be sufficient to figure out if it's the GC.

Note that this is a debug flag and thus it'll be removed from the config after one shutdown automatically. You can use adb to download and upload the config.json file that would contain this, then need to force-quit AAAAXY and restart it to sure take the new values.

[divVerent]

Also adding:

    "debug_show_gc": "true",

which will show the length of GC pauses as an overlay. This too isn't saved permanently.

[divVerent]

Of course, all this is for the Go GC; there's also a Java GC going on. Need to look into that one later.

[divVerent]

For Java GC, can you follow these instructions? https://source.android.com/docs/core/runtime/gc-debug#using_sigquit_to_obtain_gc_performance_info

[griffi-gh]

are these new flags available in the latest fdroid build?

[divVerent]

Probably not yet. 1.3.530 first has to appear on F-Droid before we can use these. Usually takes till about Friday till they have a new build.

[divVerent]

Also added debug_persist_flags that can be used to keep the debug options in the config. I'll put that on my own mobile devices once I have a build with these options on F-Droid or Play.

[divVerent]

OK, all is released now. Please put in your config.json:

{
  "debug_persist_flags": "true",
  "debug_show_gc": "true",
  "debug_profiling": "10s"
}

Then play the game normally, and cause the events that may be GC pauses for at least 10 seconds. If a bunch happened, do this:

$ adb shell ps x | grep aaaaxy
$ adb shell shell kill -s QUIT process-id-found-from-ps-command

After that, the logcat output should have some info.

[griffi-gh]

yeah gc pauses happen rarely, must be something else then.
profiling option should output some other data too right?

[griffi-gh]

hm lag is only noticable when portals are visible (and graphics are set to high/max)
more portals - more freezes ?
some rooms are unplayable while some run smoothly

[griffi-gh]

this one runs the worst (at what feels like 5 fps with constant freezes)
works fine on older versions

[divVerent]

The other data from profiling output is in adb logcat.

You'd generally ideally look for lines containing GoLog, as well as the GC dump (if possible). Note that I myself never succeeded at getting this GC dump - need a rooted device for that apparently.

Portals themselves shouldn't be too expensive, as they don't really exist ingame - however this particular section is very heavy at loading new tiles and discarding old ones (which normally should show up in the GC overlay as more frequent than usual GCing).

I still suspect there might be something going on with the Java GC; sadly no idea how to find that out without root.

Does this section still have constant freezes at lowest graphics quality settings?

[griffi-gh]

oh, so tile loading/unloading is probably the issue then?
the lag doesn't seem to corrallate with amount of portals directly, but this theory makes a lot of sense.
java gc is unlikely? I assume games like these don't have much java code at all (NativeActivity/GameActivity etc)

[divVerent]

Yes, but you've seen that the Go GC is innocent. I can never get its pauses above 0.2 milliseconds.

Tile loading/unloading is a potential culprit, and the adb log would show that if you have "debug_profiling": "10s". Can you provide the log messages? Feel free to grep for GoLog.

[griffi-gh]

yeah my go gc pauses are always <= 0.1ms and happen once per ~10s.

I'll try getting the profiling logs later

btw webgl version doesn't freeze and works significantly better overall on the same device for some reason.
also, I tested the play store builds and they have the same issue.

[griffi-gh]

Tried testing on other devices, this issue probably affects most if not all Snapdragon-based Samsung devices.

Both A52s (snap 778g) and S21 FE (snap 888) are affected

Weirdest part is that the game runs significantly worse on the s21 fe then on a52s

[divVerent]

That would actually fit the suspicion that it has to do with tile loading, as this can be rather CPU heavy.

Except that you said it depends on graphics setting - tile loading isn't influenced by that.

With the profiling logs we should be able to see more.

[griffi-gh]

hm, can't get it to work. I can get logs from adb logcat -e aaaaxy (nothing of value in there) but I don't see any tracing stuff, even after the time specified in config.
Do i have to kill it explicitly with kill -s?
(ps x/ps doesn't display any processes except sh, so idk how to get process pid)

[KiralyCraft]

I am also having the same issue on LineageOS 20, on a Galaxy S9, using the latest version at the time of writing. It runs fine in the beginning, right before jumping down into the first abyss. When on the edge, the stutter inferno begina. Also at the piano part, particularly unplayable due to very bad stutters. I did not investigate the issue yet, although I am rooted and might be able to offer insight into the matter

[divVerent]

At this point I simply have no idea how to improve it further. I guess I will keep this open with "patch welcome" - I am not an Android expert, and I also know that Android does not support non-JVM languages well and this is an example of it.

After all, the same code runs rather well on iOS.

On Mon, Jul 3, 2023, 01:44 Alex Kiraly @.***> wrote:

I am also having the same issue on LineageOS 20, on a Galaxy S9, using the latest version at the time of writing. It runs fine in the beginning, right before jumping down into the first abyss. When on the edge, the stutter inferno begina. Also at the piano part, particularly unplayable due to very bad stutters

— Reply to this email directly, view it on GitHub https://github.com/divVerent/aaaaxy/issues/223#issuecomment-1617394828, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB5NMBGEGALAIGLJQVC44LXOJL2NANCNFSM6AAAAAAYAVFBBU . You are receiving this because you commented.Message ID: @.***>

[divVerent]

In any case, if anyone can attempt this GC trace on an affected device, it could help. I can't do it as none of my phones are rooted and I can't root them without having to reinstall everything (which involves week long letter requests to banks so I can reauthenticate the online banking app). Sorry.

[KiralyCraft]

It seems to be much better now, on both a Xiaomi Mi A1 and my original Galaxy S9. I have since removed Google Applications and updated to a new LineageOS version on the S9, and the current version of the app is 1.4.42; The settings have defaulted to Auto (High) and it's all smooth now.

I suggest that people who had issues with Android performance give it a shot again, and if it's fine it might be worth considering this issue fixed.

[domportera]

It seems to be much better now, on both a Xiaomi Mi A1 and my original Galaxy S9. I have since removed Google Applications and updated to a new LineageOS version on the S9, and the current version of the app is 1.4.42; The settings have defaulted to Auto (High) and it's all smooth now.

I suggest that people who had issues with Android performance give it a shot again, and if it's fine it might be worth considering this issue fixed.

this latest update fixed the performance for me too, LG V60 running Android 13. thanks for the heads up!

[divVerent]

Thanks! I suppose this may have been done by recent updates to Ebitengine.