[PROPOSE] LiteDB in WebHost + Observables

[mbdavid]

Hi guys, this is a propose to new lib that will works with LiteDB - support for host LiteDB over websockets.

WebSockets

The main idea is host database as a server database but inside in your own web application. API should be like this:

// define entrypoint for LiteDB web hosting
app.UseLiteDB("/litedb", c =>
{
    // host web admin UI to manage your databases
    c.UseWebAdmin(); // na versão free pode ser algo bem simples

    // use custom authentication (see below)
    c.UseAuthorization<IAuthorization>();

    // register many databases as you want
    c.Register("mydb", new EngineSettings { Filename = "c:/data/mydb.db" });
    c.Register("mydb2", new EngineSettings { Filename = "c:/data/mydb2.db" });

    // or create a function to register/create
    c.Register(name => new EngineSettings { Filename = $"c:/data/{name}.db" });
});

This will server LiteDB over HTTP and WebSocket API. Will be possible to connect via normal client:

// connect to you database via web-sockets
var db = new LiteDatabase("ws://localhost:8008/litedb/mydb", "admin", "pwd123");

This websocket remote access will open possibility for another non-.NET languages to connect on LiteDB database, like node or even javascript browser;

// open database as remove address
var db = new LiteDatabase("ws://localhost:8008/litedb/mydb", "admin", "pwd123");

// get collections
var customers = db.getCollection("customers");

// run commands like insert, update, delete or query
customer.insert({ _id: 1, name: 'John' });

Run your database direct from browser javascript will be awesome!

About authentication, you will set lower level what user can to. IAutentication interface will contains:

interface IAutentication
{
    IUser Login(string username, string password)
}

interface IUser
{
    string Username { get; set; } 
    bool HasPermission (DbCommandEnum command,  string collection)
    void OnInsert(BsonDocument document);
    void OnUpdate(BsonDocument document);
    void OnDelete(BsonDocument document);
    void OnQuery(string collection, Query query);
}

With this interface, users will have a fine access to what documents and commands will be executed by this user. Will be possible check insert/update/delete data structure (use JSON.NET Schema validator) or add more required filters in queries (avoid a client read data from another client).

Observables

To offer a better user experence, will be nice if LiteDB reacts from database changes, even in remote envirments. So, the ideia is support observables to run in .NET (or any other language, like javascript).

var db = new Database("ws://localhost:8000/litedb/mydb", "user", "pwd");

var col = db.getCollection("customers");

var s = col.subscribe("CustomerId == 15", function(entity) {
    // detect any change in CustomerId == 15
});

// to unsubscribe
s.close();

[FuturistiCoder]

@mbdavid check my repos: LiteDB.Realtime I implemented Observables

[nh43de]

@FuturistiCoder @mbdavid awesome work. I initially started using LiteDb as an alternative to having a Mongo or SQLite store for very simple use cases and self-contained apps. Recently I started thinking about use cases for a real time changefeed system like @FuturistiCoder has done. Databases having such features natively is very common e.g. Cosmos, RethinkDb, Mongo, even MSSQL has them.

In the back of my mind has been LiteDB support for a Websocket API. Something like this could be very cool, but that space is also really crowded and solutions exist already. LiteDb, like SQLite, seems highly unlikely that it will be run in a centralized manner. If you need a hosted and networked SQL DB you will probably use MySql (or w/e) and not SQLite due to the tradeoff between lightweightedness and manageability. In the same way, I see LiteDB as fitting for disposable or service/grain-local temporal data stores, and not benefiting much from the bloat of full management systems. However, Websocket support would still be very cool for simple classes of apps.

Because of these reasons I think having a robust subscription/notification engine and API, and not necessarily a management or CRUD API or even websocket interface, would be of value. With a robust API in the code for notifying changes, bootstrapping communication layers on top of that would be easy, and users and bring their own stack. I see use cases ranging from IoT to reactive apps.

Some example APIs that can be learned from: RethinkDb (https://rethinkdb.com/docs/changefeeds/javascript/), kSqlDb (https://github.com/confluentinc/ksql#monitoring). I also think having good support for LINQ would be really nice.

AWESOME work I love LiteDb.

[nh43de]

If collections in LiteDb implemented something like IObservableCache from DynamicData, it could then serialize each Change object and push it over Websocket. A library like that could then be reused for other purposes besides LiteDb. I would have to do some experimenting.

[mbdavid]

Hi @nh43de, thanks for this great comments. At the first moment I thought this same question: if your wants server-side database (or cloud database) there are many options (including free options). But there is some differences in my ideia of implementation:

1) There is not separate server application running (like an external .exe) - will run over your existing web app (if the case). 2) You will have fine configure of user and permissions over each operation each user can do. Will be possible intercept documets/queries before run on database, like:

• User bob can only read collections order and customer and I any query over customer I will add this constraint: $.createdUser = 'bob'
  • User 'john' can only insert in order (and nothing more, not even query) and I will intercept this insert to set doc.custumerId = 123; 3) With this fine interception in your web app (and run over WebSocket), you can deliver your database direct from browser Javascript. And more, you can run local database (IndexDB) synced with server database. 4) Observables will run over filters (and will be possible to intercepted by server host). Clients (event javascript clients) can subscribe for changes and you can write simples web application with realtime operations direct from browser (like firebase)

The ideia with this add-in (because will not change LiteDB core - will be another lib) is provide a realtime database for web applications, storing in nosql database with programmable interceptors.

What did you thing about this?

[mbdavid]

@FuturistiCoder, I saw you lib, ver nice, thanks. I will study more your code. But I have one question, As I saw, you are notifying clients before commit, right? and if user run an rollback? I

[FuturistiCoder]

@mbdavid from my code

        public bool Commit()
        {
            var success = _engine.Commit();
            if (success)
            {
                NotificationService.Notify();
            }

            return success;
        }

it will notify only after the commit.

[FuturistiCoder]

@mbdavid since there is no ILiteEngine injection for now, I cannot adapt my plugin to your latest LiteDB. please considering merge this PR ? #1526 thanks.

[bddckr]

I wonder if this gives issues, it's doing this in a fire and forget fashion due to the Task.Run usage here:

https://github.com/FuturistiCoder/LiteDB.Realtime/blob/9d08769a07abce5419b20b0859c97cbc0890bc55/LiteDB.Realtime/Notifications/NotificationService.cs#L96

https://github.com/FuturistiCoder/LiteDB.Realtime/blob/9d08769a07abce5419b20b0859c97cbc0890bc55/LiteDB.Realtime/Notifications/NotificationService.cs#L110

[nh43de]

4. Observables will run over filters (and will be possible to intercepted by server host). Clients (event javascript clients) can subscribe for changes and you can write simples web application with realtime operations direct from browser (like firebase)

Does that mean LiteDb will have first class support for Observables notifications, like @FuturistiCoder 's lib?

[mbdavid]

Hi @FuturistiCoder, I didn't see this part in code. But how did you store documents before notify in commit? In a in-memory list? And if my transaction contains a very large data set (like 100.000 documents)?

@nh43de, yes, the ideia is implement some like @FuturistiCoder did, but inside LiteEngine class with support to send this notifications over websocket. I will take a better look into System.Reactive before start this.

[FuturistiCoder]

@mbdavid No, I don't store any document. If you updated 100000 documents, and you just want to listen to the one document, the notification service will retrieve the document for you from the db.

[FuturistiCoder]

@mbdavid there will be a performance issue if we want to subscribe to the collection with a huge mount of documents. I think I will add IQueryable filter for collection subscription.

[FuturistiCoder]

@bddckr I don't want the notification blocks the current thread.

[nh43de]

@FuturistiCoder @mbdavid let me know if you need help - I have been working with Reactive and Observables for a several years now.

[pazicb]

Hello, when do you expect the first alpha version of LiteDB for WebSocket to be available?

[mbdavid]

Hi @pazicb, I was just waiting some feedbacks about websocket implementation to starts this project. How do you want use websocket connection? It's important to us understand how dev will use this.

[FuturistiCoder]

I think Websocket could be done by SignalR. we add LiteDB as a service into a Asp.Net Core server with predefined routes / hubs to supports subcriptions of collections or documents. Authentication/Authorization could be supported by Asp.Net Core Authentication/Authorization middleware.

[pazicb]

Hello, I use LiteDB together with Remote.Linq for network projects. Both products are excellent. When I saw your concept with a websocket, it immediately caught my attention. Especially simple to use. As you describe it is great from my point of view. I would just recommend compressing the database a lot during transmission.

[pazicb]

Hello, when do you expect the first alpha version of LiteDB for WebSocket to be available?