Database Interface / DBAdapter

[techfort]

LokiJS has a combination of powerful features that are missing from most databases:

• in-memory
• self-updating materialized views (aka DynamicViews)
• can be embedded, run on node.js
• nosql

LokiJS could serve as a fast-accesss nosql-interface for any database. LokiJS would "transform" any insert/update/delete into queries for the underlying db, and load data from it. So you could get crazy stuff like a nosql interface for an Oracle DB, and MongoDB style queries for CouchDB. Any ideas on the implementation? The trickiest part I think would be the transformation of rdbms tables into collections.

[seriousme]

TL;DR: custom adapter

Full answer: Mapping different database models is notoriously hard. Fronting couchdb with Loki might not be too hard as couchdb does not maintain referential integrity between records by itself. But if you have an Oracle DB with complex schema, stored procedures etc there is imho no way you can write a coverall adapter to this. I personally like the idea of mesh (former crudlet). Just make sure the backend datasource gets a mesh adapter and there you have a nice abstraction to connect anything to anything. Alternatively one can write its own adapter to loki (e.g. using the change api or subscribing to change events) to do the right thing.

SQL and noSQL are different paradigms and mixing them imho typically results in a load of plumbing. Its already stated somewhere that administrative applications typically have 20% of their code dedicated to logic and the remaining 80% doing conversions and other sorts of plumbing.

[techfort]

well my idea is more to use Loki as a fast-access data-layer, so primarily as an extremely efficient read mechanism. I completely understand (if not TOO well) the intricacies of stored procedures, complex schemas, triggers and whatever else you can throw in, so yes, it looks like a daunting task.

But I've also learnt that the more daunting the task, the less people out there are willing to tackle it :)

Maybe if we spent a good deal of time on plumbing those conversions, developers using Loki could reverse that Pareto to 80 logic and 20 converting. Wouldn't that be doing good to the world :)

I would probably think of implementing adapters which should be straight forward or at least relatively easy, like mongo or rethinkdb.

Having said that - mesh does resolve a good deal of problems, and especially thinking of the last few ideas emerging in #6 , it may be worth considering either adding mesh.js as a dependency or re-implement it (if we don't want our software to hinge on someone else maintaining it).

[obeliskos]

So loki has ridden off of javascript's loose typing. If you are talking about integrating with strongly typed database system tables (as opposed to storing json server side) would there be any benefit from allowing json validator hooks into loki which it can validate objects inserted into it? Or would that be best left for server side to validate?

[techfort]

I would leave it to the server... but i don't know how feasible it is. You would have to introduce a middle man between loki and db, losing much of the advantage, or do i misunderstand?

My idea is to use events (basically the Changes API) and turn them into queries through drivers. So this would mean two major groups:

1. take a Changes API event and translate it into SQL.
2. the motley crew of all nosql DBs, with each "transformer" to be written as a custom adapter. Mongo should be easy, couch db feasible, rethinkdb should be easy too.

I understand the message @seriousme is giving me, and I quite agree: insert and update events are tricky at best, most likely they're an epic migraine waiting to split your skull. But if your insert logic requires complex validation and running a couple of stored procs before executing an insert, which triggers a couple of after_insert triggers, then LokiJS ain't the best fit, partner :)

Talking about strong-types: js will never be strongly typed, so you can only sanitize your code with tools that allow to check for consistency (e.g. flow), but I wonder if we could implement schema'ed collections, maybe using a fantastic tool like @gcanti's tcomb docs - which would come in handy if the underlying DB was a strongly typed one. Collections could be restricted to a schema of fields with a precise type, which would allow for better serialization and safer transforms. What say ye?

[seriousme]

Well if I wanted to build a more complex application I'd probably use Meteor anyway ;-).

Imho the sweat spot for Loki is the casual coder who needs a lightweight, good performance client side db with a not too steep learning curve. But then again I might not be your typical Lokijs down loader ;-)

[techfort]

btw - just did a little experiment as a PoC: it's a complete 2-minute dirty hack, but just to show where i was going, look at this

[seriousme]

Thats one way of doing it. Another way of doing it is like this

User still sees the standard Loki api, but on the back events are being tapped and forwarded to whoever is interested. Continuing this line of thinking I could imagine a setup where you have:

var db=new Loki(dbname)
initChangesAPI(db);
initDV(db);
initRethinkSync(db);
initWeirdIndexing(db);
initBeerTap(db);
etc..

This way the programmer can select what options should be attached. Non-present options don't take loadtime nor cpu cycles (no if's to check if they are not present). This is imho what database-inside-out is about.

Alternatively you could also create a rethinkLoki class with:

rethinkLoki.prototype.insert(record,callback){
     this.lokidb.insert(record);
     this.rdb.insert(record).run(this.rdb.connection, callback);
}

I guess its all a matter of preference ;-)

[techfort]

yes! precisely. As i said, the gist was only a hack to demonstrate the forwarding of the query :)

A side effect approach would probably lead to something like

var db = new loki();
db = initIndexes(db);
db = initRethinkSync(db);
// etc

because the function would not mutate the initial object, so it would make a copy of it, mutate it and output it, do I get this right?

[seriousme]

The events should be handled in such a way that the consumer does not modify the original object. So either the event producer does a clone ( more safe) or the consumer does ( more performant as unnecessary clones can be avoided ;-))

Side-effect-free might not be the right term as you specifically want to have side effects here :-). Decoupled might be the target here :-)

You can have parallel events (as demonstrated in my example) where all consumers get the same stream or serialized ( pipelined) events where each processor transforms the stream and its consumer only sees the transformed events.

BTW using immutability as a paradigm there is no way to do: db = f(db) it will always need to be: db' = f(db) ;-)

[techfort]

I am cooking something up - in my mind it's some amazing revolutionary thing, I will welcome your "grounding" thoughts to bring me back to planet earth, once I reveal my findings of course :)