Plugin prototype

[travisturner]

Overview

The goal for the Plugins prototype is to demonstrate how we might approach customized Pilosa queries. Use cases revolve around applying known algorithms to a Pilosa data set and letting the nodes in the cluster perform longer-running jobs which apply those alogrithms to local data and return results to an aggregator node.

The prototype should support the following example problem:

Given a standard data set spanning multiple slices, return the set of Bitmaps containing
between 10 and 20 bits set (Profiles) inclusive.

Presumably the solution will handle the problem in parallel and, like a map reduce approach, in multiple steps:

• Map: Send the map request to all applicable slices ("return the list of bitmaps with <= 20 bits set")
• Reduce: For distict bitmaps returned from each slice, sum their counts and exclude any < 10 or > 20.

Functionality Needed for the Prototype

• Implement a [Lisp/LUA] interpreter
• Modify the PQL parser to accept a [Lisp/LUA] program
  • Note: it's not a hard requirement to support [Lisp/LUA] within PQL itself, but it was suggested as a way to leverage the underlying map/reduce logic already present in pilosa.
• Provide the Plugin the ability to distribute its Map portion of the [List/LUA] program to other nodes
• Provide support within Pilosa for Scanning all bitmaps in a fragment; a Plugin might choose to leverage this.
• For simplicity, the results supported by Plugins will be the same as current pilosa results: valid results are either bitmaps, counts, or top-ns (id:count).

Future requirements

• Register a Plugin with the server via API
• Have the ability to reference registered Plugins via PQL (or an API endpoint)
• Provide protection against excessive memory or CPU usage caused by the Plugin code.

[benbjohnson]

@travisturner Go is coming out with plugin support in 1.8: https://tip.golang.org/pkg/plugin/

Do we want to consider making the plugin system in Go? It'd be nice to have the testing support and integration with doing Go.

[yuce]

@travisturner It's pretty easy to embed LuaJIT (usual caveats with depending on cgo). There's also a javascript interpreter written in Go called Otto, in case our audience would be more familiar with JS: https://github.com/robertkrimen/otto

[benbjohnson]

I agree that LuaJIT's easy to integrate. It had some caveats when I last used it around max heap size, recursion depth, & integer precision but overall it was a great tool.

Do you know what the performance of Otto is? IIRC it's interpreted and can be pretty slow compared to LuaJIT or Go.

I'd still prefer to use a statically typed, compiled language that has testing support built-in (e.g. Go). I'm taking a look integrating the 1.8 plugin system to see if it'll work.

[yuce]

Since Javascript is very prevalent, it would lower the barrier to entry, so having support for it some day would be great. Not sure about the performance of Otto, but I would guess it's much slower than LuaJIT. Since it is written in pure Go, I guess there's no reason we can't have a Go 1.8 plugin for integrating it.

[travisturner]

closing this in place of the work being done in #517