Create TODO.md

[AKushWarrior]

If you want to post a TODO.md, I'd be happy to put some time into this and the other Isar libraries.

I presume that Isar is meant to be "Hive 2.0", essentially Dart bindings to lmdb?

[simc]

Hey, Thanks for the offer. I'll add a todo asap.

Yes, Isar is essentially the successor of Hive but it is a little bit more than bindings for lmdb: Isar will support:

• (compound) indexes
• queries
• auto migration
• multithreading
• maybe relationships

All the above also needs to be implemented with IndexedDb on the web side.

[AKushWarrior]

Ah, so it's a cross-platform interface to embedded databases with some advanced features wrapped in.

That makes sense to me. On the Dart side, will it use conditional imports?

[simc]

Ah, so it's a cross-platform interface to embedded databases with some advanced features wrapped in.

Exactly

On the Dart side, will it use conditional imports?

Yes. Initially, I wanted to write the web backend in Dart but I decided to use TypeScript instead. The Dart code will glue everything together.

[AKushWarrior]

@leisim why TypeScript instead of JS? Obviously static typing is nice, but does TS play nicely with https://pub.dev/packages/js?

[AKushWarrior]

Also, it might be worth looking into something like https://dexie.org/. As far as I can tell, it has complete support for the first 2 of your features. As for the last 3:

• I'm pretty sure IndexedDB has architectural constraints which disallow multi-threading. Even if you spin up web workers and then call requests from them (which is infeasible anyways due to the cost of worker communication) the engine will just queue them. It is only able to handle one request at a time.
• Not quite sure what you mean by auto-migration. From Hive to Isar? Isar v0.x to Isar v0.(x+1)? Migration in case the underlying technology (LMDB/IndexedDB) changes?
• I assume you mean foreign key-primary key relationships. Those are not supported in this wrapper, though I think that it's very feasible with a custom wrapper.

[simc]

Also, it might be worth looking into something like https://dexie.org/.

Dexie is an amazing library. In fact, it inspired some of the architectural decisions I made with Isar. For example the "WhereClause" is a great way of letting the user decide which index to use. I still think Isar would benefit from a custom implementation because it is important to make the web & native experience as similar as possible for the user.

I'm pretty sure IndexedDB has architectural constraints which disallow multi-threading

Sorry I have not been clear: Multi-threading is only planned for the native side.

Not quite sure what you mean by auto-migration

When the schema changes (properties are added or removed), the database automatically adjusts indexes and everything else without any actions from the user. This also works with IndexedDB

I assume you mean foreign key-primary key relationships

Yes. While it would be quite easy to implement such a feature for isar-core, it is almost impossible to do it in IndexedDB with decent performance. While there exist a few JavaScript libraries which support some kind of relationships, they suffer from performance issues because of the limitations of IndexedDB. My goal is that the developer never needs to think whether the app will run on mobile, desktop, or web and have 100% feature parity. So for now I decided against relationships and backlinks but if many mobile developers request this feature, we might add it in the future.

[simc]

but does TS play nicely with https://pub.dev/packages/js?

It does. TypeScript generates JS that looks like manually written code.

[simc]

I created the todo off the top of my head. I'll extend it over time.

[AKushWarrior]

but does TS play nicely with https://pub.dev/packages/js?

It does. TypeScript generates JS that looks like manually written code.

Cool. I've written a lot of JS, but I've only really used TS in some simple Angular webapps. I'll take your word that it works.

[AKushWarrior]

I created the todo off the top of my head. I'll extend it over time.

Cool. I'll take a look to see what (if anything) I can do.

[simc]

I've written a lot of JS

Amazing, I'm not very experienced with either JS or TypeScript so I can definitely use your help.

I'll take a look to see what (if anything) I can do.

Sorry, the code is still changing rapidly and without comments. I'll work on that :+1: I hope it's not to bad and you can navigate it...

[AKushWarrior]

@leisim

Maybe I should start scaffolding out TypeScript to match the Rust layer?

The reason that using dexie would be nice is because the code is already available, minified, and low maintenance. That ways we can focus on native/Dart and leave JS/TS to a dedicated dev team; it's especially true given how much crossover the wanted features list has.

On the other hand, it's just another layer where things can go wrong which we can't fix. If I was this library's architect, I think I would use Dexie for now, and then migrate later if necessary.

I'm not telling you how to run your project; if you want me to write the TS wrapper from scratch, just say the word. I just don't know if it's the most efficient usage of time considering the end goal.

[AKushWarrior]

@leisim I probably need a decision one way or another to start on this...

[simc]

@AKushWarrior sorry for the delay. The reason is a problem / possible feature I was thinking about: isar/isar#2

It needs to be solved before we can make further architecture decisions.

[AKushWarrior]

That's right. I'll think about that one then :).

[frank06]

Happy to see you @leisim in open action again! 🎉

So for now I decided against relationships and backlinks but if many mobile developers request this feature, we might add it in the future.

If this is of any help, I see how you could focus on core features (like #2 - fantastic!) and let third party libraries implement higher level features?

Fwiw, I implemented relationships in Flutter Data using a global bi-directed graph notifier. It works with Hive as local storage, but without using Hive built-in relationships.

I'm really looking forward to having Isar as default backend.

[AKushWarrior]

If this is of any help, I see how you could focus on core features (like #2 - fantastic!) and let third party libraries implement higher level features?

Fwiw, I implemented relationships in Flutter Data using a global bi-directed graph notifier. It works with Hive as local storage, but without using Hive built-in relationships.

The problem is that this is still inefficient on web because IndexedDB has NO parallelization whatsoever. Thus, providing relationships would only be helpful on native, which goes against the stated purpose of cross platform feature parity.

On the other hand, I fully agree that some features (ex. foreign database integration drivers) should be abstracted into other libraries.

[frank06]

Interesting, I never worked with IndexedDB. So if I had a to-many relationship modeled using a foreign key, issuing a query with a where clause on that PK is inefficient?

[AKushWarrior]

Interesting, I never worked with IndexedDB. So if I had a to-many relationship modeled using a foreign key, issuing a query with a where clause on that PK is inefficient?

I had a diagram somewhere I sketched out, of sample relations and how internal architecture affects them. Let me see if I can dig it up.

[AKushWarrior]

Okay I found the diagrams, but they're more or less incomprehensible. Some notes I pulled off them are below. For the purposes of this, assume that the user starts with a predefined db which is "foreign" and "HasMany" of something. The user will now input something into a table, in which objects "BelongsTo" the foreign table.

• There are two purposes to relationships: maintaining referential integrity and performance improvements.
  • Referential integrity improvements don't really make sense in the context of a KV database, even if it has query support.
  • It's impossible to implement relationships (AFAIK) in a manner beyond the naive:
    • "take a user inputted object (belongs to some foreign group) which contains a foreign key"
    • "go find the correspondent object to that foreign key in the foreign db and then store it with the user inputted object"
  • This method is not great; it essentially balloons space usage. The reason that a better system is impossible is that:
    • we can't hold a memory pointer or reference to an object in another database in IndexedDB.
    • This is obvious; the memory address of a given object in IndexedDB varies from instance to instance on the same machine.
  • The alternative, then is to run a query on the foreign database every time we need to find the foreign information by key, which is no longer a relationship at all.

So a syntactical abstraction is possible, but there isn't really a sane way to implement "real" relationships.

[AKushWarrior]

@frank06 maybe we should relocate this discussion to #6 :)

The todo.md is done, so I'm closing this.