Problem / solution

Some users of the module need to access filtered data.

So let the user define indices, so they can retrieve data grouped by a field of their choice.

Server API

`POST /events/v1/indices`

Request Body (application/json)

Index Definition

{
    "id": "blocked-users-by-username",
    "channel": "users/v1/blocked-users",
    "field": "data.username"
}

`GET /events/v1/indices/<index_id>/<value>`

Response Body (application/json)

{
    "id": "blocked-users-by-username",
    "field": "data.username",
    "value": "testuser",
    "rows": [{ events... }]
}

Implementation

We will store the indices in Redis. I suggest we do lazy updates of the indices, in the middleware for the GET request above.

The indexer can make use of this module's own event stream. ganomede-events already keep named stream pointers internally (see clientId in the events stream getter). This way, it'll process the events it didn't already process before returning a value.

For each event in the stream, access the value for the field to index. lodash (or whatever underscore-like library we're using in this server) provides a helper to access a "deep" field by name.

Then, push the event id in a List: index:<index_id>:<value>.

Returning the list of events in the GET request is just about using this list to get all ids, then get all the listed events.

Complain to past self

Why is this module implemented with Redis!?

indexer-storage module:

createIndex(indexDefinition, cb)

addToIndex(indexDefinition, event, cb) // Add the event to the index.

getEventIds(indexId, value, cb) // Return the list of event ids

indexer-stream-processor module

Go through the list of unprocessed events for a given index, use the indexer storage module to store those in DB.

processEvents(indexDefinition, cb) // Fetch the new events since last time, feed them into `addToIndex`

// will probably use pollForEvents

index middleware

to be defined

j3k0 / ganomede-events

Event stream indices #32