checkpointing data extraction + more

[chase1124]

Hi,

As mentioned in the discussions, I am interested in a "batch" feature that would allow vuegraf to be run asynchronously without losing data or requiring user logic with --history to catch up. To accomplish this I have created a --batch command line option which does that on a per channel and per time precision basis (for each channel, example GID 123456-1, there is a separate checkpoint for Day, Hour, Minute). It knows the maximum data retention on the API and also the maximum request windows and will massage any batch requests as needed. I would also like to suggest some additional improvements (my opinion) like a "detailedDatapoint" for influxdb which has the channel type identifier (the label of circuit type) to enable more automated graphing of circuits, channel multiplier usage, etc, as well as some smaller modifications like gzipping the influx writes (better network performance and keep from influxdb write failures which i encountered using the --history CLI option) as well as using second precision on writes instead of the default nanosecond as there is no need (this is recommend by influx for better db performance and compression). Other small changes which we could discuss which I believe prevents any duplicate data between listUsage and chartUsage. All these things are done by the batch process I have created so would be easy to apply to the listUsage where desired (see below). I think the --history function could be effectively dropped as --batch is much more powerful while essentially doing the same thing (you can give it a CLI arg to override the checkpoints / initially seed the db by going back X days)

the listUsage normal operation could easily be combined with the checkpointing so that if you stop running vuegraf for sometime and then start it up with the --batch option it will catchup and then drop into listUsage collection again. I think this could be universally useful for data loss prevention and probably solves whatever problems were in the other issue about data loss as I believe the main cause of this is that the pyemvue library / emporia API sometimes only returns devices and not devices and their channels which will cause small or large data loss depending on the precision / function being executed (why I found per device and per precision checkpointing to be necessary).

I'm still testing but wondered how we can best collaborate? As I mentioned I'm no developer so I'm not familiar with practical details of modern development on commits. I guess you need to give me access to do a pull request to a development branch/tag? I just have one giant commit on a new feature branch -- some help on this part would be appreciated if you are interested in pulling the code.

[jertel]

I'm open to the suggestions. I'd like to see the implementation before merging it, of course.

The way to do this is for you to:

1. First fork the vuegraf repository. This will clone the repo into your own GitHub account.
2. You will then commit/push your changes into your repo (you can do this to a custom branch in your repo, or directly to the master branch -- your choice).
3. Then you create a Pull Request back to my Vuegraf repo's master branch.
4. I can then review your changes and collaborate further.

I don't fully understand this sentence:

I would also like to suggest some additional improvements (my opinion) like a "detailedDatapoint" for influxdb which has the channel type identifier (the label of circuit type) to enable more automated graphing of circuits, channel multiplier usage, etc,

I understand a checkpoint is a marker in time to denote an event, such as where the last data stopped updating. But your description sounds more like you want to record metadata about the device channel into the InfluxDB and that's where I'm having trouble correlating that back to the concept of a checkpoint.

Regarding the other suggestions like gzipping the data, truncating the nanoseconds, I suggest start small with the first PR and include the changes for the most important thing you need added to Vuegraf. Once we get that merge in we can move onto additional PRs. This will help minimize the changes and also helps identify bugs rather than merging in very large changes at once.

Thanks for your help improving this project!

[chase1124]

Got it, thanks for those instructions! Your suggestions for how to move forward make sense but I guess I'll have to think about how I can chunk the changes on my side into smaller PRs. I'll see how it goes.

Regarding your Q about the checkpoint vs some of the others features/changes -- you are correct that they are mostly not related to the checkpoint feature. As you say, we can discuss in smaller pieces.

This will take me a little time

[chase1124]

Just an FYI, I decided to spend a little more time working with the changes with some more datasets and scenarios to try to both catch some bugs and figure out what is most useful (also working on other stuff). I know there's no rush but just to let you know I haven't disappeared :)

[github-actions[bot]]

This issue is stale because it has been open for 6 months with no activity. Stale issues convey that the issue, while important to someone, is not critical enough for the author, or other community members to work on, sponsor, or otherwise shepherd the issue through to a resolution.