Significant Rewrite Upgrade Recovery

[phoo1234567]

Hi Douglas,

Thanks a bunch for writing this. I've been using your rtlamr-collect daemon since Feb 2018 and it's been rock solid for over a year.

Recently, it got flaky. I think it was a Fedora update that did it. So I decided to just get the latest and see if it would fix things. Unfortunately, I didn't read the change logs and things went pear shaped rather quickly.

I think I still have historical data but I'm not an Influx expert so I'm slogging through that right now which is kind of interesting in and of itself.

I was curious, though if you know of some export/import steps to just get to the new schema and keep the old data in tact. Kind of a migration document.

It's not critical but it would be nice to keep my last 12+ months of electric consumption data available.

Thanks and keep up the tweaks and feature enhancements,

Mark

[bemasher]

Can you provide your current schema? I don't recall making any significant changes to the schema, but I could be wrong.

It should be fairly straight-forward to write select statements that dump all of your data into the new schema.

[phoo1234567]

Hi Douglas,

Here is a pastebin with some database info:

https://pastebin.com/fXthxZ8w

I'm still slogging through it but it's no longer inserting new samples.

Is there supposed to be an rtlamr user?

Syslog tells me that it's trying:

https://pastebin.com/8xvHpF7R

Perhaps it's the absence of the "5m_power" query that's screwing it up. Like I said I'm swimming over my head right now. It is super interesting, though.

I'm guessing that main.go is trying to use to 5m_power as its source (which looks like the real data) and since influx is ignoring it, everything breaks down. It looks to me like the "autogen" query was the source of the "real" data in the version I was using before (Feb 2018).

Anywho... I'll keep poking at it in my spare time. If all else fails I'll just install from scratch into a new database and then figure out how to back fill the historical data into it.

Thanks for trying to help,

Mark

[phoo1234567]

Hi Douglas,

It seems that the influx measurement name changed from "power' to "rtlamr" sometime after the Feb 2018 version (I didn't look too hard as to when it happened). I use meld here and didn't see an easy diff utility for github.

It's less of a hassle for me right now to just revert the name rather than deal with migrating the influx data and adjusting the Grafana dashboard. I just tweaked main.go and did a go install and that seems to put things right again.

It also appears that the 5 minute data is an order of magnitude less than what it used to be. Perhaps you found an error while digesting the raw data from the meter? Either way, I'll probably export and re-import historical data to match the new scaling.

Thanks again for your help and your work,

Mark

[bemasher]

I went back through the schema and a few things changed, first the 1w_5m retention schedule is now just autogen since scratch space for de-duplication is no longer required. The first continuous query would need to be adjusted. Adding support for multiple meter protocols also renamed the "power" measurement to "rtlamr" because you can now measure water and gas as well. Finally because readings can come from different meters and protocols now, the multiplier mechanism was also removed.