Create Data Node Server which can compute power spectra

[BrianJKoopman]

Displaying power spectra has been a commonly requested feature request.

I believe we can achieve this through use of the plot.ly extension in Grafana. The idea would be to take compute the power spectra and return two time series with matching timestamps. These can be used in the plot.ly plugin to plot the two against each other.

In typing this out I realize this might require two data node servers, as the http server doesn't support returning two timestreams at once to Grafana based on a single query.

In any case, this might be possible, but it will take some effort and exploration.

[ahincks]

Yes, being able to display power spectra will be very useful, if not crucial. I think we should work on it.

One idea I've had since early on is the possibility of having an argument in get_data() that tells the data node server to perform some operation on the time stream before returning it. So there could be something like:

d = get_data(field, start, end, operation="ps")

(If there is downsampling, I suppose the operation could be done on the downsampled vector, but we might want to think about how exactly to do this if the downsampling isn't simply the mean value but involves some min/max scheme that can show spikes appropriately.)

If we remain restricted to SimpleJSON on grafana, we would need some way to expose this possibility. The only way I can think of is if the grafana_server provides two possibilities per field, something like dans::field and dans::field_ps. Then, when you request dans::field_ps, it does a call like dans.get_data("field", start, stop, operation="ps"); the "timeline" associated with it would would actually be a vector of frequencies called something like timeline_freq.

I think the above would be easy to implement for the near future.

Looking forward, though, this makes me think that we should seriously consider writing our own plugin for grafana that can handle this kind of thing more elegantly. For instance, there could be a checkbox or dropdown next to the field selection for selecting a power spectrum. This is something I may look into over the summer, but let's be in touch.

[guanyilun]

I think another option is to move some simple computations to the client side (in the browser) which may reduce the stress on the data node server and become more scalable. For example, the Plot.ly plugin in Grafana can be extended to compute arbitrary transformations of the time series data such as FFT. Here I modified it to compute FFT in the browser, as a proof of concept.

Here is a screenshot:

[ahincks]

That looks very promising, @guanyilun! Perhaps we could make a docker image of your modified Plot.ly so that it could easily be used with sisock: would that be the best way to include it?

[BrianJKoopman]

Perhaps we could make a docker image of your modified Plot.ly so that it could easily be used with sisock: would that be the best way to include it?

I think the best option would be to test it locally first (sorry Yilun, I didn't have as much free time this weekend as I thought I'd have), then submit a PR upstream to the plot.ly plugin.

The installation of the plugin happens upon first time container startup for the grafana container, and is configured to do so with an environment variable. It'd be great if we can just keep using this installation method, as it's pretty straight forward.

[guanyilun]

I can try to submit a PR upstream but I am not sure how long it will take to get merged and then get released. According to the Grafana docs, you can install external plugins with the environment variable too like this

GF_INSTALL_PLUGINS=<url to plugin zip>;<plugin name>

so we can host the zip file somewhere and use this to install automatically.