Feature: central event backbone in the server for analytics

[jniles]

One of bhima's central focuses is transparency. This should not be limited to any particular portion of the application, but should run throughout the design of the entire system.

We have the facility to implement real-time analytics on our server/database infrastructure easily via event emitters. Since we have a transitioned into APIs for all data CREATES, UPDATES, and DELETEs, we can capture each of these individual actions initiated by any user and display them to all users that wish to see it.

Design

I propose we implement a new library, eventd, that every controller can import and use to broadcast actions that can be captured to write to logs or send to a client via server-sent events. The API will be something like:

const eventd = require('lib/eventd');

// emit a "CREATE" event to show that something was created in the database
eventd.emit(eventd.constants.CREATE, { uuid : '...', user : req.session.user });

// emit a "LOGIN" event to show that someone logged in.
eventd.emit(eventd.constants.LOGIN, { uuid : '...', user: req.session.user });

// create a listener on /events that will send all events to the client
const app = require('express')();
app.get('/events', (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');

  // emit every event to the client
  const events = eventd.subscribe(event => {
    res.write('data: ' + JSON.stringify(event));
    res.flush();
  });

  // remove the listener when the client kills the connection
  res.on('close', () => eventd.unsubscribe(events)); 
});

If we had this infrastructure, you can imagine the ability to quickly add event streams for user login/logout events, database operations, reporting, etc. All of these could be available to power feeds on the client to give real time information about the system.

[jniles]

After some discussion with @sfount, this proposed feature accounts for real-time analytics, but does not have a storage mechanism to see "events from today" or "events this week".

To account for this, I propose storing events in a database table event that looks like this:

timestamp	user_id	type	data
2016-05-12 12:19:40	2	CREATE	{ table : "patient"}
2016-05-12 12:23:21	7	UPDATE	{ table : "user", id : 2 }

... where data is a JSON column that could store variable information.

With this new proposal, the client would connect to two server routes:

1. /events will simply return a list of events to the client, limited to the last day by default. It will be downloaded once.
2. /stream will be a real-time stream of events, published via server-sent events.

An additional limitation of this model is that events cannot be shared across cores (since the event emitter will only exist in a single process. Eventually, we could move this event emitter architecture to a shared database like redis. See this example for more information.