RSS balloons to multiple gigabytes over multiple days

[Atemu]

Steps to reproduce

1. Have the app running on a phone with lots of memory (8GB in my case)
2. Be in a handful of somewhat active rooms
3. Observe RSS after multiple days

Outcome

What did you expect?

RSS should be a few hundred MiB at most.

What happened instead?

It's a few GiB.

Your phone model

OnePlus 5

Operating system version

Android 11 (Homebrewed LineageOS 18.1 + F-droid + µG + Zygisk)

Application version and app store

1.3.18, olm 3.2.10, F-droid

Homeserver

matrix.org

Will you send logs?

Yes

[Atemu]

Another bug I thought was separate is insane storage usage. It used 8GiB of "user data" before I killed it and then went back to just 260MiB.

Might just be some sort of GC when restarting Element, will check when it happens again.

[ouchadam]

to double check, is the data usage you're referencing data storage, rather than mobile data usage?

for the storage side of the issue, it's a "feature" of the realm database we're using, there's some more information here https://github.com/vector-im/element-android/issues/5078

[samueldr]

I'm observing this as RAM usage, Element is dwarfing the next biggest RAM user, and as disk usage. Disk usage gets reset just like for @Atemu here, from 4+GiB to 3xxMiB on force close / relaunch.

I don't have a good setup to track RAM use, but it will end up causing apps to hitch and hiccup all over, and force closing/relaunching solves the issue for a while, as @Atemu reported. I observed the RAM usage through System → Developer options → Memory → Memory used by apps.

[ouchadam]

@samueldr thanks for clarifying :100: does your account have access to any rooms with large member counts?

[samueldr]

I don't know how large is large. Maybe. One room I know of has 2787 members. I follow other more active rooms, but bridged using "non-puppeting" bots, e.g. one member writing loads and loads of messages relaying messages from other protocols.

So still "maybe", it depends how large is large. A handful of them.

[Atemu]

Interesting seeing you here @samueldr :D

We are both in the Nix community and share a set of rooms, so that's a connection. We are both in the Nix/NixOS room which is rather large and active for example: https://matrix.to/#/#nix:nixos.org

We also both use robotnix-built Android images (I presume) but that would be a rather odd cause.

[samueldr]

Yeah, I can confirm I'm running it on a Robotnix-built LineageOS ROM.

I recall getting an update notification from F-Droid recently, and I don't recall experiencing any of this before:

[removed wrong part, seeing @Atemu had the issue with an earlier version.... and uhh... I too faced the issue with the version released before 2022-02-13.]

It might not be the latest update that caused the regression, but a recent one

[Atemu]

It just happened again on 1.4.0.

Did you just come here because it happened to you too?

[samueldr]

(Sorry, I was heavily editing my reply)

I don't think it's the latest updates that are at fault, but maybe recent update(s) still. I don't know for which update it would map to, if my observations are correct.

[ouchadam]

I don't know how large is large

sorry I should have clarified, I would consider a large room to be one with more than 1000+ members

---

to clarify the issue, unexpected amounts of memory are being used by the element app? as for versions it sounds like at least 1.3.18 and 1.4.0 are affected.

have you noticed the memory eventually freeing up by itself when the usage starts to peak or does it always require a force stop? (or potentially the system killing the app).

[samueldr]

I don't have good tooling to track things. My observations rely on the behaviour of the system and other apps.

When the behaviour gets bad, the system and other apps start lagging/hitching. For example, but not limited to, playing an emulated PS1 game lags in spikes. It comes and goes, as if whatever causes the hitches (assumed to be Element) maybe worked to evict some memory usage. But then it comes back maybe 10 minutes later. Before noticing it was Element, nothing I would do would help. Since noticing, force closing and not restarting Element makes the hitching go away. Starting it back also doesn't make the hitches come back quickly, I have no observed time span it takes to start acting badly again.

And yes, the developer menu shows high RAM usage from Element, 3+GiB. Next highest "app" is system with ~1GiB use.

Phone is Redmi Note 8 Pro (begonia) with 6GiB of RAM, reported as 5.9GiB by Android.

[ouchadam]

thanks for the info :+1: I'll try setting up a test device, joining a selection of large rooms and leave element running in the foreground for a while

[bkchr]

Hey, I also have seen similar things. At maximum I have seen a 4.9gb memory usage of the app. My system also gets very laggy and I need to kill the app. I'm also part of multiple big rooms.

[Atemu]

@ouchadam if you've already got a device running, you should be able to see the RSS explosion by now because I just did.

@bkchr I can see you're also a Nixpkgs maintainer, are you in the Nix / NixOS room too? Does your device run on a robotnix-built image?

[bkchr]

I was also in the main nixos room. I left it now and it seems to have improved my memory usage, but as I'm still in other big rooms the memory usage is still relative high. I'm running the default image from Samsung for my device, so nothing special there ;)

[ouchadam]

my testing setup - Nexus 6P Lineage 15.1 / Android 8.1.0, F-Droid variant. new account with the following rooms #matrix-hq, #megolm-test and #element-android

I've noticed leaving the app in the foreground causes the memory usage to continuously grow, I was seeing about 10mb~ per hour, however when putting the app into the background (at least on my device) the size eventually shrinks back down, I saw my usage go from 1.7gb to 230mb~

[Atemu]

@ouchadam good to seee you're able to reproduce.

Could you try again with the Nix / NixOS room perhaps? That's the one we've all been experiencing issues with.

[ouchadam]

joined :+1: (along with the large genshin impact room)

IDLE IN FOREGROUND

judging from the real time memory graph (which shows the usage go from 320mb~ to 500mb~ over the course of 10 minutes), I'm guessing we're either constantly appending data to our in memory caches (without limits) or there's a memory leak which stops us recollecting memory

EDIT: more findings...

MEMORY	ALLOCATIONS	ROOMS

Seems like the native realm allocations are the cause our increasing memory usage :thinking:

• logs attached
• my device is also very warm, not sure if this is due to the activity of the rooms or the memory usage causing increase CPU usage

[MurzNN]

@ouchadam Can you please explain how do you create such graphs in Android? I have the same problem, but can't find any way to build chart with app memory usage history...

[ouchadam]

@MurzNN these graphs are coming from the memory section of the profiler within android studio or intellij, you'll need to be running a debug build in order to attach the profiler to the running element process

[kolaente]

I'm observing the same behavior since joining the full fossdem space with a lot of rooms and sub-spaces. I'm also in the nixos room but didn't see this before the fossdem space.

[sebschrader]

I've experiencing this issue since at least 1.3.18 (F-Droid build) and I'm currently at 1.4.0. The biggest room (measured by number of people) has 500 people in it, but a very low number of messages sent per day.

Maybe related: The disk_store.realm file is also growing to massive size (multiple gigabytes). Force-stopping and restarting compacts it again down to a few mega bytes (See #5078), but after a few hours it has grown again.

stat /data/data/im.vector.app/files/$hash/disk_store.realm
  File: disk_store.realm
  Size: 3552485376       Blocks: 6938576         IO Blocks: 512 regular file
[…]

[kj]

Not sure if I'm adding much here, but I've been seeing this on my OnePlus 8 (Element 1.4.2). My system has been becoming unresponsive lately and I finally decided to check memory usage today and Element was sitting at 2.9GB. User data (storage) was also at over 6GB, and after restarting Element that went right back down to 28MB.

I'm not in any public channels though. I simply use Element for two channels: a 'Note to self' channel and a 'Prometheus alerts' channel which I haven't even set up yet (but with which I intended to create a bot for notifications). Obviously neither have any real activity, and I barely open the app once a day.

[samueldr]

I would hazard a guess that the fact you're not in any rooms with activity or large population does add to this issue.

[ouchadam]

@kj thanks for the information, this is quite interesting! are you running the fdroid or google play variant?

the difference is important because the fdroid version never truly idles~, it will poll the homeserver every X seconds (defaults to 60s, each poll is at least a sync request to the homeserver), which makes your idle/low activity case interesting because low activity between each poll/sync would mean (ideally) the app does almost nothing

however, the google play variant should only wake up and sync whenever a push is received (a new message being received or marking a message as read on another client/session)

---

FWIW, for my personal use case, a handful of low activity, low member count private chats, with the google play variant I rarely see the app reach above 300mb~

[kj]

@ouchadam I'm pretty sure I installed the F-Droid version. Perhaps I should try the Google Play version, particularly as it sounds like it might be easier on battery life anyway?

[ouchadam]

a quick way to tell if you have the f-droid version is if there's a persistent notification which mentions listening for notifications, or in the Settings -> Help and About -> Version will contain a G- for google or F for f-droid

if you're able to switch over then the google play variant will improve battery usage, there's also the possibility that the system will be able to safely destroy the app in the background to reclaim memory with push messages recreating the app when needed (the fdroid build can't do this as it must always be running in order to poll the homeserver for new events)

I also notice your user data is quite large which makes me suspect you're encountering #5244

[kj]

Ah! Yes I do have that persistent notification (and I did wonder about that, as changing the background sync settings didn't affect it). I also checked the version and yes, it is the F-Droid version so I'll give the Google Play version a go.

[neutralinsomniac]

Google play version exhibits the same issue

[kj]

Since installing the Google Play version I'm now not even seeing Element in the memory usage list, so presumably in my low usage case that means that it's now being closed while inactive.

[Aang23]

I also have this issue here. (OnePlus 6T running Lineage 18.1, F-Droid only with no google services or other addons).

Until a few weeks ago, all was fine, but now over the course of a few days RAM usage increases until Element becomes absolutely unusable, and sometimes results in my phone swapping enough to freeze for a few minutes.

Force-stopping Element once in a while fixes the issues entirely.

I've checked over ADB as well, memory usage matches.

Honestly, I don't mind Element using 1-2GB that much, I don't do much else on my device, but it's certainly annoying to have it impact the app's usability.

Edit : Overall, I am not in any very active or huge rooms, mostly smaller groups on my homeserver. Usually RAM usage was under a GB, and my usage has not changed since this issue appeared.

[Atemu]

FWIW, I can't remember the last time I saw this happen. @ Nix people, were any of you hit by this bug recently?

[Aang23]

I still am experiencing the issue as of now, despite a few updates in the meantime.

[Atemu]

I haven't seen this happen since I switched to unified push. I'm not 100% sure it did happen before the switch or version update that enabled it but I'm not experiencing this bug anymore.

I think we should we close this and if somebody else is able to reproduce this, they should open a new issue with their specific circumstances.