Define Meteor API

[badslug]

How does this package integrate with Meteor? How does a Meteor developer use the data/connection?

I'm thinking something like creating a top-level Streams object in the package (assuming the ddp server is used to track real-time streams and is probably in a Meteor package named 'streams'). The Streams.Stream object is a parallel to the Meteor Mongo.Collection. Developers use Stream on the Meteor server or in the browser to either create a reactive variable (similar to Mongo.Collection.findOne) that tracks the real-time current value of data point or the real-time updating statistics document (average, min, max, etc).

Stream can also be used to create a Mongo.Cursor (similar to Mongo.Collection.find()) with time series data for a data point over time. The ddp-server is responsible for pulling and updating those reactive values and/or cursors (using DDP connection to the Meteor server) similar to how the MongoDB driver works.

Code example:

var device = ... // reactive device document (from Meteor MongoDB collection)
var str = new Streams.Stream("foo"); // A real-time data stream of "foo"

// The following calls take a `selector` as first parameter - either a Mongo Selector, Object ID or string
// just like Meteor.Collection.find() - here we assume device.id is an object ID or string

// From client we can then add into a template a real-time tracked data point `datum`
// which is a Meteor reactive variable
var datum = str.trackData(device.id, "temperature"); // current temperature of device with matching id

// From client we can add into a template a real-time tracked data statistics `stats` document
// which is a Meteor reactive variable
var options = {
  include: [
    'average',  // average using default configuration
    { name: 'min', interval: [-7*24*60*60, 0] } // minimum over the past week
  ]
};
var stat = str.trackAverage(device.id, options, "temperature"); // temperature stats of device with matching id

// From client we can addd into a template a data graph based on 'timeline'
// which is a Meteor Mongo.Cursor
var options = {
  interval: 5*60*60, // Interval between time slots in seconds - 5 minutes
  range: [-(7*24*60*60),0], // start and end window for data points - 5 days to now aka the past week
  // Alternatively we could set `period` which would be absolute days if we wanted to show calendar week
  ... // other options?
};
var timeline = str.trackTimeSeries(device.id, options, "temperature"); // temperature over time of device with matching id

Alerts?

To parallel what most people use real-time streaming databases for, an additional feature would be to have the ability to "alert" based on data in the stream. This could be very complicated as it will probably require injecting code into the streaming server or creating a query language. Assuming we can inject limited capability Javascript into the streaming database something along these lines could be possible?

var str = new Streams.Stream("foo");
str.watch(function(context, datum) { 
  if (datum.temperature > 100 || datum.temperature < 0) {
    Session.set('alert.'+datum.id, true);
  }
});

How would the function run on the streaming server but still access Meteor objects like Session? Probably not practical? Could we instead do some sort of query language like a Mongo Selector (aka a Stream Selector) that calls a registered callback on the Meteor server when it matches... or maybe when it changes match state? Or could we use javascript for the selector but restrict what objects it can access to just say a 'context' object which we pass in and has a limited API (maybe stateless with a limited in-memory cache)?

Anyway, if we could do something like this we could keep most/all the business logic in Meteor but still allow a real-time streaming database to monitor and trigger actions efficiently (without having to pipe all the data into Meteor so the meteor server has to process all the data.

This could be the longest ticket I have every created.

[scoward]

Congrats on the longest ticket :).

For alerts, I think that the user would just use the value (whatever we return it as) from the trackDate function and surround it with Tracker or similar sending out alerts if the value is invalid (or whatever). I don't think a watch function that send executable code to the server is something that we want. If they really want alerts in real time without the web page open then they should use Kinesis IMO.

[scoward]

var datum = str.trackData(device.id, "temperature"); // current temperature of device with matching id

@badslug What is 'temperature' there for? Is that info sent to the server to select the variable that's sent back?

[badslug]

The primary reason for setting alerts via the API is that devs on the Meteor side can define alerts that run on the infrastructure rather than on the Meteor side. If meteor devs create alerts on the server then the infrastructure has to send all data to the Meteor server to be processed. We want to avoid churn on MongoDB and alerts help avoid churn on Meteor server.

[badslug]

Yes, temperature would be the name of the datum to extract. So in this example, device data streaming past, and the selector device.id would select only data that belongs to the matching device, and that we want the temperature data point.