MMKV storage file grows until app crash

[ryskin]

The library is great but one crucial issue doesn't give me chance to use it.

And it grows by stop 64Mb, 128Mb etc. If I use slow FS storage, the file size never becomes more than 15Mb. I also used the console log to check the size of the string.

It's working really fast and I want to use that, but with every rewriting, it becomes more significant. I checked key is always stable.

"react-native-mmkv": "^2.4.3", "react-native": "0.66.3",

[mrousavy]

How often are you writing to MMKV? It's not a database, it's a memory map key value storage.

[ryskin]

How often are you writing to MMKV? It's not a database, it's a memory map key value storage.

I'm writing when my state change with a few seconds of throttling. I use it with redux-persist. But why does the size of the file grow? As I mentioned before, I use FS storage, which is significantly slower, but the file size increases up to 15Mb. I use only one key.

[ramirobg94]

i have the same "bug"

Check the explanation here https://github.com/ammarahm-ed/react-native-mmkv-storage/issues/286

Seems that the Key Value doesn't overwrite the value, just add a new row into the storage. @mrousavy

Please @ryskin let me know if you solved the problem

[mark-careplanner]

Any further thoughts on this? We are seeing a similar issue

[mrousavy]

Possibly related? https://github.com/mrousavy/react-native-mmkv/issues/397

[aLemonFox]

@mrousavy to clear things up, does mmkv insert a new row for every setter, without overwriting the previous value? If so, how can we properly clear 'old' entries?

[frozencap]

@aLemonFox have you tried removing the previous entry before setting the new one?

[JacobFJ]

@mrousavy Confirming this contributes to my app crashes. The memory is hogging in Android 12 Go edition which typically 2GB or less ram.

[ramirobg94]

@shawarmaz this is dangerous because you can get into two cases:

1 -> async delete, then async write -> maybe write is too slow so there is no advantage to using mmkv 2 -> race condition -> The new data can be deleted

I see the point on how this memory is designed but should have some cleaning method to reduce the memory consumption, there is no point in keeping dozens of versions (Is there any method to recover previous versions? I think that no, so it is pointless)

[frozencap]

@ramirobg94 neither of those cases apply because the calls are sync. To be clear that means not async.

[mrousavy]

This is apparently how Tencent/MMKV is designed.

[arthurgeron-work]

Checking if it exists and deleting before set seems to have worked; I've used patch-package, but I can open a pr with the fix:

diff --git a/node_modules/react-native-mmkv/lib/module/MMKV.js b/node_modules/react-native-mmkv/lib/module/MMKV.js
index 8856aea..9cbe764 100644
--- a/node_modules/react-native-mmkv/lib/module/MMKV.js
+++ b/node_modules/react-native-mmkv/lib/module/MMKV.js
@@ -45,6 +45,12 @@ export class MMKV {
   }
   set(key, value) {
     const func = this.getFunctionFromCache('set');
+    const contains = this.getFunctionFromCache('contains');
+    if (key && contains(key)) {
+      const _delete = this.getFunctionFromCache('delete');
+      _delete(key);
+    }
     func(key, value);
     this.onValuesChanged([key]);
   }

You could also just leave the delete call there before set to avoid increasing the time this function takes to finish:

diff --git a/node_modules/react-native-mmkv/lib/module/MMKV.js b/node_modules/react-native-mmkv/lib/module/MMKV.js
index 8856aea..11b43da 100644
--- a/node_modules/react-native-mmkv/lib/module/MMKV.js
+++ b/node_modules/react-native-mmkv/lib/module/MMKV.js
@@ -45,6 +45,10 @@ export class MMKV {
   }
   set(key, value) {
     const func = this.getFunctionFromCache('set');
+    if (key) {
+      const _delete = this.getFunctionFromCache('delete');
+      _delete(key);
+    }
     func(key, value);
     this.onValuesChanged([key]);
   }

[frozencap]

I don’t think this should be merged, maybe just a doc update. You can just wrap the method yourself.

[arthurgeron-work]

I don't think this should be left as is, having it fill up memory is a serious issue, having it go a little slower is much better than having it crash your app.

[frozencap]

Again, this is how mmkv is designed. You can wrap the method yourself. Wrapping it lib-level would give new non-mmkv semantics to the lib. To keep it 💯 mmkv, the actual solution to this issue is mmkv’s trim method.

@mrousavy , what was the issue with this PR?

https://github.com/mrousavy/react-native-mmkv/pull/461

[arthurgeron-work]

I don't know how not managing space allocation could be by design? Managing memory/space allocation is the libs responsibility, otherwise it's a insta leak to unaware devs. There should be a big warning in the readme, a temporary fix like that can be given but the way data is stored needs to be changed to fix the leak, or at least trim should be exposed and it could be called in specific situations, even though this delegates responsibility to devs and they can't really tell when trim should be called because they can't see how much space has been allocated.

[frozencap]

Read this https://github.com/Tencent/MMKV/issues/610

[arthurgeron-work]

So it's either delete and trim manually or move to another lib like react-native-mmkv-storage, gotcha. Another thing is that it seems like trim isn't a exposed method

[ryskin]

I don't know, I started to use other MMKV libraries and problem disappear, file is small and works like charm

[codering]

I don't know, I started to use other MMKV libraries and problem disappear, file is small and works like charm

which lib？

[ryskin]

I'm thinking not good to post links to other libraries here, but you can google only 2 main libraries available for react-native

[arthurgeron]

I don't know, I started to use other MMKV libraries and problem disappear, file is small and works like charm

which lib？

react-native-mmkv-storage works just fine

[mrousavy]

@mrousavy , what was the issue with this PR? https://github.com/mrousavy/react-native-mmkv/pull/461

@shawarmaz I found out that printing the total size after doing a trim made no difference, so I wasn't able to reproduce your error. I could take another look, but we wanna find the right solution here without sacrificing speed :)

Can you guys create a reproduceable example? Is this on Android only, or iOS too?

[frozencap]

Personally I just wrapped writes with preprended deletes and called it a day. No noticeable difference in write performance.

If you're testing against high volume of writes or high write size, you're not using the right tool for the job. The official Tencent/MMKV repo benchmarks it against NSUserDefaults/SharedUserPreferences which tells you all you need to know about its intended usage i.e. not as a local runtime database.

If you have a relatively small collection of key-values that you'd like to save, you should use the SharedPreferences APIs.

The NSUserDefaults class provides a programmatic interface for interacting with the defaults system. The defaults system allows an app to customize its behavior to match a user’s preferences.

IMO the main selling point of MMKV is not speed but having a unified API for use cases that would have otherwise required NSUserDefaults/SharedUserPreferences.

If you want to store application data like chat messages or POI geopoints etc, use the right tool for the job i.e. something else. Even if trim was available everybody would be like wtf why do I need to do this. It's counterintuitive because you think MMKV is something that it is not.

This may be a case of false advertisement around MMKV at large. Thoughts welcome

[aLemonFox]

@shawarmaz I think you're right. I'm struggling with this too. Are there any other good reactive key value store libraries available which are not based on this? I can't seem to find any. They are either like AsyncStorage (async, not reactive) or WatermelonDB (full db, want key value store like Redis).

[frozencap]

@aLemonFox just use this lib and wrap writes with a preprended delete

  setItem: (key, value) => {
    storage.delete(key)
    storage.set(key, value);
  },

...or just unsexy AsyncStorage whose writes do overwrite and that also does not load entire data from cold storage in memory, unexpectedly

Which in fact also illustrates that comparisons/benchmarks between AsyncStorage and MMKV are inherently false/misleading.

[frozencap]

@aLemonFox Btw mmm MMKV is not reactive

[arthurgeron-work]

@aLemonFox Btw mmm MMKV is not reactive

Personally I just wrapped writes with preprended deletes and called it a day. No noticeable difference in write performance.

If you're testing against high volume of writes or high write size, you're not using the right tool for the job. The official Tencent/MMKV repo benchmarks it against NSUserDefaults/SharedUserPreferences which tells you all you need to know about its intended usage i.e. not as a local runtime database.

If you have a relatively small collection of key-values that you'd like to save, you should use the SharedPreferences APIs.

The NSUserDefaults class provides a programmatic interface for interacting with the defaults system. The defaults system allows an app to customize its behavior to match a user’s preferences.

IMO the main selling point of MMKV is not speed but having a unified API for use cases that would have otherwise required NSUserDefaults/SharedUserPreferences.

If you want to store application data like chat messages or POI geopoints etc, use the right tool for the job i.e. something else. Even if trim was available everybody would be like wtf why do I need to do this. It's counterintuitive because you think MMKV is something that it is not.

This may be a case of false advertisement around MMKV at large. Thoughts welcome

It's not only falsely advertised, it's broken by design. Moving from async to mmkv is like switching to a bmw but that suddenly crashes into a lamp post if you push it too hard.

Anyway, we need to test to be sure that prepending delete to write calls does actually fix the leak.

[arthurgeron-work]

@aLemonFox Btw mmm MMKV is not reactive

I think local storage should not be reactive, for that it's better to actually use something like Zustand or Redux and persist that data on specific moments, either with a specific solution or something like mmkv

[frozencap]

It's not only falsely advertised, it's broken by design. Moving from async to mmkv is like switching to a bmw but that suddenly crashes into a lamp post if you push it too hard.

Anyway, we need to test to be sure that prepending delete to write calls does actually fix the leak.

@arthurgeron No it's not broken by design, it is your understanding of its design that is broken. It is extremely efficient at what it is designed for, surely everyone touting its benefits aren't idiots. Did you even read my reply you quoted? Going from AsyncStorage to MMKV when you have a big load of high-write data is like trying to park a schoolbus in your garage.

I think local storage should not be reactive, for that it's better to actually use something like Zustand or Redux and persist that data on specific moments, either with a specific solution or something like mmkv

Read the context before replying please

[arthurgeron-work]

Regular use of a lib that does not describe itself as "not production ready" should never cause a crash regardless

[frozencap]

It is definitely production ready. For some unknown reason, you just don’t want to understand what it’s for.

Stop spreading such misinformation and communicate constructively instead of ranting and complaining @arthurgeron

[arthurgeron-work]

Please explain how I am spreading misinformation, so regular but intense use of this won't generate a crash? Writes don't create new allocations and leave old ones hanging?

[frozencap]

@arthurgeron I have already explained many times it is not meant to be used that way. For the last time, MMKV is meant to be used as a replacement for NSUserDefaults and SharedPreferences. Not as a database

[arthurgeron-work]

Tencent MMKV used as a base for this lib works in a way that wasn't clearly advertised, sine clearly RN MMKV intended to be used to store info like it was done to Async Storage, which causes the crash, still, not safe to use in production, even with workarounds, because it by default uses it as a "DB"of sorts which it isn't intended to do in the first place.

I'm not going to reply on that subject any longer, reasons for that should be beyond clear given your passive-agressive and vulgar vocabulary, which only gets in the way of a clear respectful discussion.

[frozencap]

Glad things are finally clear.

This does require an update to the docs tho.

@mrousavy if you agree with what has been said here about MMKV usage, I'll draft up the doc PR

[arthurgeron-work]

Shouldn't be just a update in the docs, most people that use this lib are unaware, in favor of avoiding crashes on thousands of project's I'd prepend delete to set by default, if that is proven to work; On top of that I'd place a warning in postinstall so everyone is aware.

[mrousavy]

@shawarmaz 100% agree with your comment.

[mrousavy]

It's not only falsely advertised, it's broken by design. Moving from async to mmkv is like switching to a bmw but that suddenly crashes into a lamp post if you push it too hard.

@arthurgeron-work @arthurgeron what? lol that does not make sense. Your understanding of what MMKV is for is wrong, it's not broken by design. I also mentioned that it is a key value storage a couple of times, I have compared it to other key value storage like solutions but that does not mean they are exactly equal and only differ in performance.

That's like comparing an Audi with a Ferrari, they are just not built for the same purpose.

[mrousavy]

On top of that I'd place a warning in postinstall so everyone is aware.

I think this is a terrible idea for multiple reasons:

1. No one reads that.
2. Warning people about a lib after installing it is the wrong place to do so. The docs (README) should explain what this library is for, and if people use it wrong it's on them.
3. What exactly should the warning be? That you should not spam a small key value storage with loads of data?

I am providing the code I wrote for free for anyone to download - that code runs in my apps in production without any problems and faster than any other library - if you use it correctly you benefit from it. If you use it wrong, call it false advertisement or "broken by design", then don't use it and don't comment here :)

[mrousavy]

Now let's stop with spamming hateful comments - I have a productive idea: A fastWrites: boolean prop to the MMKV Config (constructor) that basically controls if writes should skip deleting to write more efficiently or not. We can leave it false by default, and if users want more speed they set it to true.

That way I don't get any annoying comments of people misunderstanding the reason why MMKV exists.

[arthurgeron]

It's not only falsely advertised, it's broken by design. Moving from async to mmkv is like switching to a bmw but that suddenly crashes into a lamp post if you push it too hard.

@arthurgeron-work @arthurgeron what? lol that does not make sense. Your understanding of what MMKV is for is wrong, it's not broken by design. I also mentioned that it is a key value storage a couple of times, I have compared it to other key value storage like solutions but that does not mean they are exactly equal and only differ in performance.

I did not meant to sound offensive, that whole conversation was unecessary drama. Still, how would it not be fair? You've advertised specifically as a replacement for async storage, which is exactly the kind of usage that leads to crashes with this lib, hence my previous example does make sense.

I understand the "issue" is in Tencent MMKV, but having a "just let people know" attitude instead of searching for a fix ( like creating a modified Tencent MMKV base ) isn't right, 2.6.x already brought a silent crash to older Android RN projects and now this is yet another silent issue that keeps hurting the trust of devs on this lib. The minimum would be to add a change (if possible to 2.5.x) as well that avoids this problem entirely.

Warning people about a lib after installing it is the wrong place to do so. The docs (README) should explain what this library is for, and if people use it wrong it's on them.

README's hardly get read by devs that have had it installed for some time and it's a huge issue for many projects, and devs have already being misled to use it like async storage so baffles me that it "it's on them". I don't think warning them in every way possible is such a terrible idea.

[arthurgeron]

Now let's stop with spamming hateful comments - I have a productive idea: A fastWrites: boolean prop to the MMKV Config (constructor) that basically controls if writes should skip deleting to write more efficiently or not. We can leave it false by default, and if users want more speed they set it to true.

That way I don't get any annoying comments of people misunderstanding the reason why MMKV exists.

Sounds perfect, that'd fix the issue on current projects, I'd leave fastWrite "false" by default and explain the "dangers" of enabling it in the docs. Also adding this fix for 2.5.x would be great

[KingAmo]

i see a explain by MMKV in Chinese, see here

you can optimize by this:

-(BOOL)append:(NSData*)data {
    if (space >= data.length) {
        append(fd, data);
    } else {
        newData = unique(m_allKV);
        if (total_space >= newData.length) {
            write(fd, newData);
        } else {
            while (total_space < newData.length) {
                total_space *= 2;
            }
            ftruncate(fd, total_space);
            write(fd, newData);
        }
    }
}

[arthurgeron]

So they did foresee that space allocation issue.. Interesting, seems like a better solution

[xmflsct]

I am curious did anyone manage to run some benchmarks? How much slower it gets with and without fastWrite?

[harjotmalhi94]

What is the recommended solution if we are limited to version 2.5.1 due to breaking changes in 2.6.0?

The changes here do not seem to do the job and I can still see multiple copies of the same key in the Documents/mmkv file

[mrousavy]

Okay since no one else did it, I created a minimal reproduceable example:

const storage = new MMKV()

for (let i = 0; true; i++) {
  console.log(`Writing "test":"Some Value in here." to MMKV for the ${i}th time.`)
  storage.set('test', 'Some Value in here.')
}

This runs for a while. After writing for the 1.270.708th time, memory size reaches 500MB. So writing 1 million times to the same key makes it grow to 500MB of memory.

Yes, that sounds like an issue. But this is a core MMKV issue, apparently they don't truncate the file after a few writes. I think this is due to DEFAULT_MMAP_SIZE being very large. I'll investigate this if I have some free time.

Since this is an issue with the core MMKV C++ library, this is also an issue with the other rn mmkv library (react-native-mmkv-storage).

I'm wondering if this is only happening in simulator due to more RAM being available? IDK what DEFAULT_MMAP_SIZE / pagesize is

[harjotmalhi94]

Thanks for testing the limits @mrousavy

I am attempting to patch version 2.5.1 with the same delete you did inside set here: https://github.com/mrousavy/react-native-mmkv/pull/509/files

But it still results in the old entries staying there and new ones being added. Any idea why the same patch in the older version does not work? Even prepending the set with a delete does not fix the issue:

setItem: (key, value) => {
    storage.delete(key)
    storage.set(key, value);
  }

[frozencap]

Here's what the main developer of Tencent/MMKV at WeChat has to say:

MMKV won't increase to an unacceptable big size under normal usage. Before each growth, MMKV will always try to pack existing data to see if there's enough space for new commer.
However, if there's an unreasonable big value stored inside once, and if there's not enough space for that big value, MMKV might grow to an unexpected large size.

https://github.com/Tencent/MMKV/issues/610#issuecomment-752015762

As I mentioned earlier, WeChat benchmarked MMKV against NSUserDefaults and SharedPreferences. Everybody should be referred to those to understand what MMKV is intended to be used for, high up in the docs. IMO this is the only part that needed to be fixed, otherwise people will keep trying to use MMKV as a database.

IMO fastWrites was a faux-pas, the naming implies one way to use it is for high volume of writes. However, the very fact WeChat ran benchmarks on something built for use cases that are not performance sensitive (storing auth token, color preferences, locale) is most likely the source of all this confusion.

And please don't fool yourselves, WeChat does not use MMKV as its database. It developed and uses a database for that: https://github.com/Tencent/wcdb

The whole point is if you even start hitting these issues in the first place with MMKV, chances are you're using it wrong.