Research: supporting using hive in multiple isolates

[themisir]

Why?

People are using Hive in background isolates to work with persistent data, and because hive doesn't have proper isolate support, and dart doesn't have proper thread support, it's currently impossible to make sure different hive instances can sync up with each other when mutating database. So proper isolate support was requested multiple times on hive/hive issues.

When?

I don't know, it totally depends on our findings on this research thread.

Limitations

• We can't send lots of data through isolates, even tho isolates are improved a lot, it's still costly a lot for workloads like database
• Dart doesn't have proper (fully compatible) Thread API
• We have to keep existing API and database structure as stable as possible

Other references

• Some comments on dart-lang threading support issue: 1, 2
• IsolateNameServer

[TheCarpetMerchant]

Hey ! @mtc-jed here.

Here's a possible implementation that didn't pan out, but it taught me a lot about Isolates. It uses dart:ui's IsolateNameServer but would need a real mutex for security in deciding which Isolate Hive lives in. To note : I wrote this in while Dart 2.15 was in beta I believe, with some flags activated for constructor tear-offs and Isolates Groups. These things are now on by default, so this actually might work as is.

I had trouble with the Isolate not responding, possibly due to the way Isolates are handled (Isolate Groups could have fixed this issue) and the permissions they have (for file access ; you can't access the app's files if the Isolate wasn't created by the app, again probably a thing now handled by Isolate Groups).

I wanted to use this to be able to access Hive from the Isolate spawned by Android's AlarmManager. It would be worth asking that team if that Isolate is part of the same Group as the main app's Isolate in all circumstances, as I believe this to be the crux of the problem with that use case.

Anyway, here goes : https://github.com/TheCarpetMerchant/HiveIsolateInterface/blob/master/HiveIsolateInterface.dart

Had to work around Hive a bit to make it work. Hope the comments make it understandable what's going on.

[themisir]

I have a few concerns about your implementations. For example:

1. IsolateA spawned from notification or alarm manager and created master hive instance
2. User opens notification, OS opens app and main / UI isolate sees that there's a alive hive port so uses it
3. OS decides to stop notification since user tapped on it and notification cleared, isolate gets killed, but hive still thinks there's master hive instance on another isolate.

So to solve that hive might need to periodically do health checks to decide which isolate has to process data which might cause race condition again (if both isolates tries to be master isolate at same time).

---

Btw I was actually thinking about utilizing ports as like event bus. So instead of sending data to master hive instance, when a hive instance makes some change it notifies other hive instances to reload box (or mark that box as dirty so next time user needs to read something it has to be reloaded). But still this has race condition issue, which could be solved using mutexes.

[TheCarpetMerchant]

What you're describing is exactly why we'd need to talk to the team handling the AlarmManager plugin, so we know how these things are behaving. I thought about calling the isAlive function before every call but that would be tremendously costly.

The other approach you're describing is very interesting, and might be easily implemented at least for LazyBoxes. I believe LazyBoxes only read the keys and corresponding offset to the data in the file ? So you'd only need to re-read that.

The real problem is how to implement a correct mutex for this in Dart. Could it be possible to handle this in Java instead ?

[themisir]

The real problem is how to implement a correct mutex for this in Dart. Could it be possible to handle this in Java instead ?

Yeah that's another option. Actually I thought about writing whole communication part (comms between hive instances) on low level language (to provide cross-platform support). We'll also not gonna have limitations on dart like shared memory space or handling race conditions. But I don't know if it does worth it. It'll probably create another set of issues.

[TheCarpetMerchant]

I think a first goal should be to provide multi-Isolate support for existing uses (ie Flutter apps), if it happens to be feasible without too much trouble, as this would unlock many use cases for Hive, such as the AlarmManager stuff. But if it happens to be too much trouble handling this in Dart itself, side-stepping Dart's limitations through dart:ffi is the only viable option. Maybe a solid Rust library could provide safe shared memory management ?

[themisir]

But if it happens to be too much trouble handling this in Dart itself, side-stepping Dart's limitations through dart:ffi is the only viable option. Maybe a solid Rust library could provide safe shared memory management ?

Yeah, I think that's why @leisim used rust instead of dart for writing core of isar.

[TheCarpetMerchant]

A simple solution could be to load the entire Hive file as-is in Rust-managed shared memory. Then instead of Hive reading/writing things to the disk, it's writing to that shared memory through Rust, with Rust handling mutexes. I'm not sure how easily you could "update" the Hive instances from there though. Maybe it's better to have a specific Box type for this shared memory system, where objects are read and deserialized from that shared memory every time we need it ?

[TheCarpetMerchant]

I'm going back to this issue since I've got some time.

Currently, I have a few boxes that are shared between multiple isolates. What I'm doing is opening the boxes, reading the data and immediately closing the box (on all isolates). In the extreme majority of cases, this causes no issues because you won't access the box on multiple isolates at the same if they aren't storing that much data. The read process will be quick enough. This has proven to work as I have thousands of users but extremely few (single digits per month) issues with this.

However, a great improvement for this would be using a shared mutex (probably via locking a file specific to this purpose) and only reading from the box when the isolate has the lock.

Just wanted to hear if you had any thoughts about this before working on it, @themisir . I've had a few times where the backend told me that "the box has already been closed" or something to that effect. Is there a way of guaranteeing Hive has released the box to prevent this error ?

This is of course just literal band-aid, but would allow secure cross-isolate usage with the current implementation. I'll do a proof-of-concept by upgrading my current wrapper class used for multiisolate purposes first.

Edit : I'm currently using this class.