Create sample for specific time

[magao]

I'm currently implementing a plugin to monitor my new solar inverters and smart meter. I can't get the data in real-time from the inverter or meter - I have to get it from my energy provider's portal which is usually around 30 minutes behind (on the plus side, using the portal means I have access to generated, consumed, exported and imported energy details).

The data I get back from the portal includes a timestamp (in UTC). Is there any way to specify that the particular sample occurred at that timestamp, rather than "now"? If not, do you see any value in adding this functionality?

BTW technically one of the inverters has real-time stats available via wireless, but it looks like I'm going to have to reverse-engineer the protocol (it appears to be a variant of the protocol handled by https://github.com/XtheOne/Inverter-Data-Logger, but UDP plus other changes), so I'm using its portal as well for now (but it's generally within a minute of real-time).

[ktsaou]

Hi,

Unfortunately you can't feed netdata with old data. Its internals are tightly integrated to now and it automatically adds gaps to the charts when data collection plugins get behind.

We could probably have some kind of batch import at some time that could do what you want.

Thankfully my inverter had an API (raining day):

[magao]

The approach I've taken is to save a JSON or CSV file to a known location in the NetData web directory (specified via configuration file), and in my custom dashboard I've hooked into state.updateChartWithData to retrieve the file, manipliate it to ensure it's in the right format and set it as state.data.result.data. This gives me the entire days results that are available. Not pretty, but it works. Might be worthwhile considering providing an API for an alternative source of data points for a rendered chart. I've personally added a data-url attribute specifying the URL (and a data-prepare attribute which specifies a function used for transforming the data e.g. from CSV to labels + data).

Note that I'm also inserting the latest values into NetData even though they don't match the timestamps as I use that for the badges.

<div
    data-netdata="Inverter.60aad3fe_ba87_45bc_80ed_943fc517ce6f"
    data-dimensions="capacity|exported|imported|generated|consumed"
    data-legend="no"
    data-chart-library="dygraph"
    data-dygraph-dimensions-hidden="capacity"
    data-url="/json/60aad3fe_ba87_45bc_80ed_943fc517ce6f.json"
    data-prepare="prepareCsvData"

I've also done a few other things so some values show as line series, others as area, etc.

From left-to-right (BTW it's fairly overcast today):

Exported to grid - green area Imported from grid - red area Generated by on-grid solar - blue line Consumed from on-grid inverter - orange line

Generated by solar attached to hybrid inverter - blue line Reported as "to meter" from hybrid-inverter - green area Load on hybrid inverter - orange area Battery charging (negative)/discharging (positive) - purple area

BTW the battery charge gauge normally is about the same height, but I've made the whole energy <div> vertically resizable and expanded it so the graphs are clearer.

Having 10kW of solar is nice. Unfortunately, it looks like something hasn't been configured/installed correctly as the battery never discharges ... it's supposed to do that before importing from the grid. Also the hybrid inverter (with the battery) is meant to be off-grid (not allowed to have more than 5kW attached with single-phase power), but it's reporting as on-grid and most of its power is reported as going to the meter, and the reported load has never been above 100W ...

Now ... if only I could have a semi-realtime API for my gas usage ... but no smart meter :(

[ktsaou]

nice!

Check google charts. Actually netdata was originally developed with google charts (check http://your.netdata.ip:19999/old/), but I switched to other chart libraries because they are way too slow.

You can find jfiddle examples of using google charts with netdata at https://github.com/firehol/netdata/wiki/REST-API-v1

I think google charts support reading CSV. Dygraphs support CSV too: http://dygraphs.com/data.html#csv

netdata also supports more chart libraries (check http://your.netdata.ip:19999/dashboard.html for a showcase).

[magao]

I originally thought I should just be able to feed CSV to dygraph, but it didn't work. I tried various things, but eventually settled on producing json as the simplest option.

BTW I want to stick with Dygraph for a couple of reasons, not least being that they're the prettiest available in NetData ;)

I've just finished a few more modifications, including charting the inverter temperature on a second axis on the same chart as the battery charge. I've managed to do everything without modifying the dashboard.js (so my changes should work with NetData upgrades). Getting the second axis working correctly was difficult as naively doing an extra updateOptions() with the second axis settings resulted in the second axis actually overdrawing an already-drawn graph. To get it to draw correctly I needed to ensure that the y2 axis was configured in the first updateOptions() after the graph creation.

When I've got the time (maybe over the new year break) I might try to pull the various hooks I've worked out into pull requests for dashboard.js. I've also managed to implement an equivalent of data-easypiechart-barcolor for the gauges (data-gauge-gaugecolor) and the ping and battery charge series change colours depending on values (and highlighting the series the colour depends on the highlighted point value). Mousing over the title displays the current series info (including hidden series) and highlighting a point shows the values at that time (with a hook to do some renaming - see screenshot for the result).

This is my current dashboard which demonstrates most of the things I've done (I'm not worried about the displayed IP addresses - all private range and firewalled).

[ktsaou]

nice work!