The future of Hive

[simc]

TLDR: Hive 2.0 will be rewritten in Rust to get amazing performance, multithreaded queries, read and write transactions, and super low memory usage. The code will work 1:1 in the browser.

Situation

I have thought a long time how to correctly implement queries in Hive and I have come to the conclusion that it is not possible with the current architecture. I have reviewed many projects on GitHub which use Hive and most of them have to create their own suboptimal workaround for queries. Apart from queries, Hive has another problem: Dart objects use much RAM. Since Hive currently keeps at least the keys in RAM, you can hit the OS limit of mobile devices quite fast.

I also created polls some time ago on the Hive documentation page and there were two very strong takeaways:

1. Queries are something almost every user wants
2. An overwhelming majority (86%) of users don't mind breaking changes

Idea

So here is what I have come up with: I will completely rewrite Hive in Rust. I will use the strengths of the old implementation (like auto migration) and fix the issues. On the VM, Hive will use LMDB as backend and on the Browser IndexedDB. The VM implementation will provide the same features as IndexedDB to allow easy code sharing. The two main goals of Hive will stay the same: Simplicity and Performance.

I have a small prototype and the performance is amazing. LMDB has to be some kind of black magic.

Sample

Here is how it is going to work:

The model definition is very similar to current models:

@HiveType(typeId: 0)
class Person {
  @Primary
  int id;

  @HiveField(fieldId: 0)
  @Index(unique: false)
  String name;

  @HiveField(fieldId: 1)
  int age;
}

Hive will then generate extension methods so you can write the following query:

var box = Hive.openBox<Person>('persons');
box
  .query()
  .where()
  .nameEquals('David')
  .or()
  .nameStartsWith('Lu')
  .filter()
  .ageBetween(18, 50)
  .sortedByAge()

where() vs filter()

The difference between where() and filter() is that where() uses an index and filter() needs to test each of the resulting elements. Normally a database figures out when to use an index itself but Hive will provide you the option to customize.

There are multiple reasons for that:

1. This code will work 1:1 with IndexedDB
2. You know your data best and can choose the perfect index
3. The database code will be significantly easier

Things to figure out

• How can auto-updating queries be implemented efficiently?
• What are the restrictions for shipping binaries on iOS?
• Should there still be a key-value store?

Blocking Issues (pls upvote)

• dart-lang/sdk#37022
• dart-lang/sdk#37355
• flutter/flutter#46887

Other issues

• WebAssembly/reference-types#61

For existing apps using Hive 1.x:

I will continue to keep a Hive 1.x branch for bugfixes etc.

What do you think?

[mdrideout]

I'm not sure. I see the pros for creating a new project but I think the new database will be very cool and it would be sad if the old "Hive" would be popular while the new version has only few users due to its bad pub.dev popularity score...

I wouldn't worry about the popularity score a lot as a decision maker on that, I have seen many similar plugins have high popularity scores. When you search "nosql database" on pub.dev, there are several options in the high 80's and 90's. I think they do a good job showing relevant options, and it seems more like relative popularity trends, rather than a fixed descending order of popularity.

If you do the rounds of getting it out there, having good docs and tutorials, it will pick up speed. The thing that attracted me to Hive was your high quality documentation. In comparison, Sembast's documentation is kind of lack luster and not nearly as newbie friendly.

[deadsoul44]

Here is my vote:

The new version should be a completely separate package with a new name. It will already benefit from the popularity of Hive. We will advertise it everywhere.

Hive should be maintained here.

Otherwise, it will be complete mess in the community. People will get confused whether if they are talking about v1 or v2 or vXXXX or sql based or nosql based, etc, etc, ....

[simc]

@deadsoul44 I think you are right. I did not think about the countless upcoming questions about the two versions and which one to use. It will probably be easier with two packages.

I will probably need some help maintaining both packages.

[themisir]

@leisim I could help to maintain this version.

[AKushWarrior]

@leisim noticed you've archived a few Hive packages. Does that have anything to do with the new package?

[yringler]

This repo is now a mono repo, and hive_generator and hive_flutter where moved into it in subfolders.

On Wed, Jun 24, 2020, 12:05 AM Aditya Kishore notifications@github.com wrote:

@leisim https://github.com/leisim noticed you've archived a few Hive packages. Does that have anything to do with the new package?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hivedb/hive/issues/246#issuecomment-648570388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7B3KKKUODPEIF7HKOOIVDRYF3SLANCNFSM4K56YKVA .

[simc]

Just wanted to let you know that I made some progress regarding "Hive 2.0".

The new database will be called "Isar" and you can check out the current state at github.com/isar. Some parts are still missing to make it usable tho.

[mdrideout]

@leisim - have you looked into adding more maintainers for the old hive? I submitted a PR for a fix a few days ago that broke the generator on the latest version of flutter. https://github.com/hivedb/hive/issues/364

[simc]

@mdrideout Not yet. Would you mind if I added you as one of the maintainers?

Who else has a little spare time? ;)

[AKushWarrior]

There's three sides to maintainership:

• project management
• bugfixing
• feature implementation

I have time for any of the three, but thought that I might point that searching for a specific category (as opposed to a broad "maintainer") might be more feasible.

[simc]

@AKushWarrior

I would be interested in any of the three :+1: Especially bug fixing and publishing new versions is required currently I think.

[yringler]

https://github.com/hivedb/hive/issues/246#issuecomment-636697140v

I could help to maintain this version.

@TheMisir , still up for it? Where you added as a maintainer?

[themisir]

@yringler yes, I could help to maintain hivedb.

[MarcelGarus]

@leisim I'm also interested in maintaining, but mostly project management / issue triage.

[simc]

@AKushWarrior @TheMisir @marcelgarus I invited you to the Hive organization. I hope I didn't forget anyone.

Also, I'd like to hear your guys options on isar/isar#2

[JagandeepBrar]

I would also be interested in helping maintain the project. I could help with bugfixing and issue triage, but would not have much time currently for feature implementation (but could change in the future!)

[themisir]

@leisim , how can we publish new versions to pub.dev? For example hive_generator and hive packages needs an update to fix dependency issues. Pub.dev latest version is currently unusable in latest flutter version due to those issues.

[JagandeepBrar]

@TheMisir He will need to add you as a contributor/member to the publisher on pub.dev. After that, you will be able to run dart pub publish.

[AKushWarrior]

Just pub publish is fine. No need for dart.

But yeah... We'd need publish permissions to be autonomous. Thoughts @leisim

[medmedmedic]

Super cool, slow queries are the limitation to build large apps in a flutter "Absolutely true". Waiting for Hive 2.0

[omidraha]

1. Can you give an estimate of when Hive 2.0 will be published with query [and sync] support ?
2. Can we use Hive 2.0 with flutter similar to hive_flutter for cross platform supporting in Android and IOS ?

[yringler]

@leisim, any timeline when new maintainers will be able to publish updates to hive to pub.dev?

[themisir]

Please help me by writing what's currently high priority issues. I'm planning to work to fix minor and major issues to make this package stable again.

Also, I'll publish the package with different temporary name (eg: hive-db or something like that) until @leisim gives us permission to publish packages to official hivedb pub.dev package.

[yringler]

@mdrideout , in regards to sembox - I understand that everything is loaded into memory? Which is not suitable for all projects. My data comes shipped in a nice sized chunk of JSON, so I prefer having hives lazyboxes (which only have to load the keys into memory)

With Hive, I was writing manual toJSON and fromJSON

You can automate this with json_serializable with hive, also.

[AKushWarrior]

That's... good for you? I understand the complaints about the maintainership situation, and I have my own gripes with @leisim, especially in his inactivity with regard to project management. On the other hand, it feels inappropriate to randomly hijack an issue about the future of a package to tell people about how another package is better.

[simc]

I just added @TheMisir as uploader :+1:

If leism had produced production apps for clients using hive, I am sure he'd have more skin in the game and would understand the issues the community is suddenly facing.

I have 6 production apps using Hive ;)

[omidraha]

Exactly the same as me, After this issue, I also migrated from Hive to Sqflite.

[simc]

@omidraha C'mon this is not the place for such posts.

[AKushWarrior]

Btw @omidraha Hive 2.0. (Isar) will be platform independent and natively work on both Android and IOS.

[themisir]

I just updated packages. I hope this will solve dependency issues. Please let us know if something is not working again by opening an issue.

[tr-stan]

thank you @leisim for providing permissions to @TheMisir, and thank you TheMisir and others for your work! I know a few posts on this issue may have seemed a little much or off-topic, but I do want to emphasize they were relevant to the future of Hive, because: I am sure there are many others like myself on small startup teams, pretty new to the field and wanting to help maintain but not knowledgeable/confident enough to do so, and seeing months go by without any resolution to some pretty frustrating issues, it could lead a lot of people to drop Hive as an option and maybe even write it off for future consideration.

Even when there were others merging in PRs to fix numerous issues, those weren't available in the more widely/easily trusted medium of pub.dev because of a lack of either communication or who knows what. Too many potential users would have been hit with pretty high walls of entry into learning/using Hive, and for someone like myself it also discourages the idea of contributing once I even feel good enough, or smart enough. Of course in this field you need to be resourceful and learn each day how to surmount those walls, but I think having at least TheMisir, who is currently capable of actively addressing current issues and publishing to pub.dev is a great step to making the current state (and even more the future state) of Hive more accessible to everyone, regardless of their skill level.

Anyway, thanks again for all y'all have done! I see now that Hive 1.4.2 and other dependency updates have been pushed to pub.dev, and at least I ran them without any conflicts with my set of dependencies (including characters ^1.0.0), without having to reference the latest state of master branch in github directly in my pubspec.yaml. Such a happy moment!

Hopefully sometime in the future when I'm more experienced/confident and not fighting every day to get my first product into the world, I'll be able to contribute to Hive too. And I hope thanks to this step others will be encouraged to continue experimenting with and investing time into Hive. :)

[simc]

@tr-stan Thanks for your opinion in this matter. To be clear, I don't want to censor any criticism. I agree that the lack of updates has not been optimal.

But I still think that this is not the right place to discuss how to replace Hive with something else. After all, this issue is called "The future of Hive" and not "The future without Hive" 😉

Edit: I'm also very happy and thankful that @TheMisir and others will continue development for now :+1:

Edit2: Don't hesitate to open a PR just because you think you don't have enough knowledge. Maybe you can start with small things like typos in docs etc.

[surajprasad]

@TheMisir @leisim looking for suggestions for my use case, I am looking at creating an asset tracking app where a user may have an average of 200 assets of different kinds. The box structure fits perfectly and the encryption provides for safety of the asset information too.

I am keen to know if I write an implementation with the latest stable version of hive (@TheMisir thanks for maintaining it) will the boxes be readable by Isar etc. in future. (What I mean is, whether objects written now can be accessible for read-only operations with future technologies).

Another noob question is whether data objects / boxes are transferable as it is to another device and be read on a new device given the same app and encryption keys, in other words are backups and restoration easy using hive. My use case does not need the boxes to be synced to a central database, they only need to reside in one place, signatures / pointers to the boxes are used in all other cases.

Thanks

[themisir]

I am keen to know if I write an implementation with the latest stable version of hive will the boxes be readable by Isar etc. in future.

I think this will be possible but a bit challenging. You have to read data from hive box and manually save to isar. Think like reading file from XML file and saving to JSON file. I would recommend adding some sort of storage abstraction layer between UI/logic and hive/isar. In that way you don't have to modify all the code when you change storage instead you will just modify code on that layer.

Another noob question is whether data objects / boxes are transferable as it is to another device and be read on a new device given the same app and encryption keys, in other words are backups and restoration easy using hive. My use case does not need the boxes to be synced to a central database, they only need to reside in one place, signatures / pointers to the boxes are used in all other cases.

You may transfer hive file and encryption key or I would suggest to manually export data to for example json file and read it when importing on other device. Because the data might corrupt (check Murphy's Law) if hive version mismatch between A and B devices. Also JSON is human readable format. The customer wont lost access to assets if data corrupts somehow.

[surajprasad]

I think this will be possible but a bit challenging. You have to read data from hive box and manually save to isar. Think like reading file from XML file and saving to JSON file. I would recommend adding some sort of storage abstraction layer between UI/logic and hive/isar. In that way you don't have to modify all the code when you change storage instead you will just modify code on that layer.

Is the current work on Isar taking into account the interoperability of objects created with the current hive db? I hope there are clear documentations to manage this transition for projects under development that use hive in the current status.

You may transfer hive file and encryption key or I would suggest to manually export data to for example json file and read it when importing on other device. Because the data might corrupt (check Murphy's Law) if hive version mismatch between A and B devices. Also JSON is human readable format. The customer wont lost access to assets if data corrupts somehow.

While JSON is excellent, I am still not very familiar with it and I am not too sure if I would use it since I want the data to remain encrypted, visible only to the authenticated users, if they choose to, otherwise a seamless transition to the new setup would be excellent. I guess I am trying to make a decision to stay with Hive and port to Isar when convenient, mainly due to the simplicity of its implementation. Great work!

[yringler]

I am still not very familiar with it and I am not too sure if I would use it since I want the data to remain encrypted, visible only to the authenticated users,

dart support for JSON is pretty good. You would want to create a class which could hold whatever data you currently have in hive. If passing around unencrypted JSON is a concern... you could encrypt it. There are many encryption options available in dart.

[surajprasad]

dart support for JSON is pretty good. You would want to create a class which could hold whatever data you currently have in hive. If passing around unencrypted JSON is a concern... you could encrypt it. There are many encryption options available in dart.

Thank you, thats helpful. Is there any clarity yet on the timeline for the release of a stable Isar candidate? thinking if I should redraw my timelines.

[simc]

@surajprasad It will take a long time until Isar can be considered stable. Also, it will not replace Hive. There are still many use cases for Hive and we (currently mostly @TheMisir) will continue development.

[surajprasad]

@leisim Thank you so much for that, as for my use case, hive in its current form is good enough, and if it will continue to develop / be maintained, thats all I need to proceed with it, my concern was whether this will become obsolete and hence I would be forced to change to a different library. Thank you, and wonderful work so far, may the force be with you!

[RaghavKaushal]

@leisim I'm planning to use hive in my flutter app, any idea when will hive 2 rollout?

[AKushWarrior]

@leisim I'm planning to use hive in my flutter app, any idea when will hive 2 rollout?

It's pretty far out; there's no hard estimate, but at least a few months for the full release.

[AndiDog]

What's the plan for web support? WebAssembly support isn't in all relevant browsers yet. Just started experimenting with hive v1 with Flutter to be cross-platform compatible.

[AKushWarrior]

What's the plan for web support? WebAssembly support isn't in all relevant browsers yet. Just started experimenting with hive v1 with Flutter to be cross-platform compatible.

It uses JS interop to wrap IndexedDB on web.

[dgandhi17]

@AKushWarrior few months meaning 2-3 months ? or rough estimate.

[simc]

@dgandhi17 Hopefully we can publish a beta version in 2020. You can expect a stable version late Q1 2021.

[tanzilzubair]

thank you @leisim for providing permissions to @TheMisir, and thank you TheMisir and others for your work! I know a few posts on this issue may have seemed a little much or off-topic, but I do want to emphasize they were relevant to the future of Hive, because: I am sure there are many others like myself on small startup teams, pretty new to the field and wanting to help maintain but not knowledgeable/confident enough to do so, and seeing months go by without any resolution to some pretty frustrating issues, it could lead a lot of people to drop Hive as an option and maybe even write it off for future consideration.

Even when there were others merging in PRs to fix numerous issues, those weren't available in the more widely/easily trusted medium of pub.dev because of a lack of either communication or who knows what. Too many potential users would have been hit with pretty high walls of entry into learning/using Hive, and for someone like myself it also discourages the idea of contributing once I even feel good enough, or smart enough. Of course in this field you need to be resourceful and learn each day how to surmount those walls, but I think having at least TheMisir, who is currently capable of actively addressing current issues and publishing to pub.dev is a great step to making the current state (and even more the future state) of Hive more accessible to everyone, regardless of their skill level.

Anyway, thanks again for all y'all have done! I see now that Hive 1.4.2 and other dependency updates have been pushed to pub.dev, and at least I ran them without any conflicts with my set of dependencies (including characters ^1.0.0), without having to reference the latest state of master branch in github directly in my pubspec.yaml. Such a happy moment!

Hopefully sometime in the future when I'm more experienced/confident and not fighting every day to get my first product into the world, I'll be able to contribute to Hive too. And I hope thanks to this step others will be encouraged to continue experimenting with and investing time into Hive. :)

Taking from @tr-stan 's reply, I'm also a newbie, and I chose Hive because it was simple to understand , and coincidentally fit my use case as well.It helped me a lot, and I'd be only too happy to be able to give back to it, when I can find some free time.

I was thinking about starting with the already very good docs, maybe clearing some stuff up and making it even more newbie-friendly than it already is. I'm a beginner, and as such confused about how I should go about doing that. Is that something I'm gonna need to do pull requests for, or is it on the docs website?.

Also, if there were simpler issues or features, I'd be glad to help as much as I could with them. Can't just let @TheMisir carry the entire thing on his back, ya know?

EDIT: I'm good with planning and stuff too, so I could be of help in issue tracking and prioritizing the major and problematic ones.Anyway I can help, I'll try my best.

[misterfourtytwo]

[simc]

@Tanzil7274 Help in every shape or form is definitely greatly appreciated. @TheMisir does an amazing job maintaining Hive singlehandedly but I'm sure he doesn't mind a little help ;)

If you want to improve the docs (which is definitely a good idea especially to make Hive easier for beginners), just create a PR in the docs repo. You'll instantly see a test deployment and once the PR is merged, the website will be updated automatically.

If you have further questions regarding organization or contribution, just ask :+1:

[Hamza-Shewa]

hi everyone I've just started using Hive DB with a flutter web app, I've created a POS system and I realized after completing the whole development process that hive stores data in the browser Indexed DB, now I'm afraid of something happening and deleting the whole DB, and this is a problem since you can't create a backup from your DB! any solution for this or should I just dump the system and start over again

[themisir]

hi everyone I've just started using Hive DB with a flutter web app, I've created a POS system and I realized after completing the whole development process that hive stores data in the browser Indexed DB, now I'm afraid of something happening and deleting the whole DB, and this is a problem since you can't create a backup from your DB! any solution for this or should I just dump the system and start over again

You could use Box.toMap() and export all data into map then convert it to json (maybe using json_serializable) to save it. Or you can use javascript to export data from indexed db which I think it would be much easier to do.

https://stackoverflow.com/questions/17783719/import-and-export-indexeddb-data