Create and archiver data saource

[shroffk]

There are various use cases where it would be convenient to access archived data as a PV. The web runtime, the save restore service, etc...

I was thinking of creating a new datasource in core-pv which would retrieve the data from the archiver like sim://noise(-10,10,1)

aa://pv_name(time_instant,...) aa://pv_name(start_time, end_time,..)

Instead of individual applications and services needed to understand how to retrieve and consume archive data.. they could use the archive data source with via a the core-pv

[shroffk]

initial feedback from jembishop

I quite like your idea of using the "url" (aa://) to specify this rather than as part of the Java API as you can use it in a phoebus widget.

You should also be able to specify a timedelta from now rather than a absolute time. Eg: aa://pv_name(now, now - 1day)

In all cases the data returned should be timestamp, value pairs.

In general there should be support to write your own java plugin to query a different timeseries store. We have an influx set up we might want to use, or potentially timescaledb instead of the epics archiver appliance. Also variables which are not EPICS based eg MQTT won't work with the archiver at all.

Similarly the prefix, (in your case "aa") should be able to be associated to the archiver implementation. If you have pva://hello and mqtt:://hello these are separate variables and will have different archival methods (eg mqtta://).

[kasemir]

Well, does it fit the PV idea? The data consists of [ { timestamp, value}, { timestamp, value}, { timestamp, value}, ...]. Report that as a table VType? This would allow showing it in a table widget, but not in any other widget. That type of PV would send just one value update. It doesn't fit the subscription idea, and is always read-only. Overall, that's certainly not a good fit for a PV.

[kasemir]

Or do you mean "data returned should be timestamp, value pairs" as a stream? When you subscribe to the PV, you get a rapid flurry of for example VDouble updates, one for each {timestamp, value} pair read from the archive? In Probe, you see rapid changes and then the most last archived value at the time of retrieval. In the data browser, it would actually plot those received values over time with their respective time stamp from the pair.

[shroffk]

So for the save restore... the use case is to get the a value from the archiver in some cases when a snapshot is not available.

In this case there would be a single value ( similar to a pvget ) aa://pv_name("closest_value_available", instant )

This would be a single update type PV.

For visualization cases where we need more than one value we can consider either a stream or table ( value timestamp ). I was initially thinking more of a table, but this could be determined by the client creating the archiver PV.

aa://pv_name("stream", now, last_week ) ==> returns each individual update... this would be used by a client like probe aa://pv_name("table", now, last_week) ==> returns table, which would be used by apps that understand how to consume them

Depending on if the time range is relative or not these could have periodic updates.

I think of these along the lines of loc:// PV's which are also relatively stable, but I agree we should give some thought to how to best support something like "caget/pvget"

[kasemir]

Got it. "Closest_value_available" is thus the initial use case. It fits quite well to a read-only PV. The rest could be added later.

[georgweiss]

Is closest available value guaranteed to be "non-null"? What if it's time stamp is years away from wanted instant?

[shroffk]

Is closest available value guaranteed to be "non-null"? What if it's time stamp is years away from wanted instant?

So, we need to decide the behaviour and document it as part of the API for this datasource. We could depend on the AA for the behaviour it adopts or have preferences that can be tweaked.

Currently, I am still in the requirements gathering stage so these questions and scenarios are great input.

[kasemir]

"Closest available value" might be a bad choice. The basic idea of the channel archiver, RDB archiver and archive appliance is that they store changes with the original time stamp. When you ask for data at a specific time, we're looking for the last sample at-or-before that time. The archived sample may be quite old for the case of operator set points that simply have not been changed in a while. We don't repeat such a non-changing value with new time stamps say every hour because sometimes you really want to know when the value last changed, and "3 month ago" can be exactly what you want to know. Any sample after the requested time, representing a change after the requested time, is not valid at the requested time. In case of channel disconnects or archiver shutdown, that's supposed to be logged as invalid samples with something like "disconnected" or "archive off" in the text/message.

So this archive PV needs to support an "at-or-before" type of request, you look for the last sample just before that time, and then you either have a) nothing in the archive at all -> no value b) an invalid sample with some error in the text -> may report that so user sees invalid alarm and message c) a somewhat recently timestamped sample -> great d) a sample with a very old sample -> you may wonder if that's correct, but that would be perfectly fine for a set point type of channel

[georgweiss]

For my use case "at-or-before" is a must. But... If the "at-or-before" value cannot be trusted, would it make sense to pick the preceding sample if that is valid?

[shroffk]

at-or-before that time

Yes, this is the behaviour to be expected from most archivers anyway.

If the "at-or-before" value cannot be trusted, would it make sense to pick the preceding sample if that is valid?

I would say no, the one instance of "invalid" data could be due to situations which could mean that the samples before then are not really valid. Imagine you had a major issue with your motor controller, hence a disconnect value, you might have to re calibrate after you get it back and now the setting from before are not necessarily valid.

[georgweiss]

@shroffk, does that mean @kasemir's case b) does not apply?

[kasemir]

If the sequence of samples is [ 1, 2, 3, disconnected-or-off ] and you're looking for a time after that last sample, then your data is that disconnected-or-off sample. Can't pick the "3" because that looks better.

[shroffk]

thanks...I guess examples without motion controller are better to understand :)

case b) is completely valid. The aa://pv_name(...) would return something with an error msg and invalid alarms. In our case the client ( save restore client) would have to handle this case...by showing some info to the user, not restoring the value, etc...

[georgweiss]

My user's case is "what was the set point value at this point in time"? If the at-or-before value is "disconnected or off" because the IOC was off-line, then 3 would be the value their looking for.

[kasemir]

If after "disconnected or off" there is no value, then you don't know the value. Can't simply pick some older data.

If the IOC recovers or the archive is turned back on, the data will look like this: [ 1, 2, 3, disconnected-or-off, 3 ]

So now 3 is your latest sample, it's valid. Its time stamp shows when we re-connected, it no longer reflects when the operator last changed it, but that's the best we can do since the archive engine does need to write a sample after that disconnected-or-off sample was written, with a new time stamp.

[georgweiss]

Can't simply pick some older data

Since I suspect my users will dispute this: why not? They want to know a set point value at a point in time when some system was running just fine, but for which they did not create a snapshot. The point in time specified by users may be just after "disconnected-or-off", simply because they do not know that the actual time they should be using is between 3 and disconnected-or-off.

[kasemir]

Your users need an editor where they can modify what's in the archive. Don't have a value, don't like the value? Make something up and enter it.

[shroffk]

I am going to repeat my example from above

I would say no, the one instance of "invalid" data could be due to situations which could mean that the samples before then are not really valid. Imagine you had a major issue with your motor controller, hence a disconnect value, you might have to re calibrate after you get it back and now the setting from before are not necessarily valid.

[kasemir]

.. which is one reason why we use an RDB archive: Sometimes a key system for recording beam information was off, we can derive the data by other means, so we back-fill that data to get uninterrupted long-term reports. Back when we checked, that wasn't available for the archive appliance, but in principle you could of course create a tool to patch the data files in a similar manner.

[georgweiss]

@kasemir, it's already possible to edit values before a restore operation, so we just need to indicate that sample cannot be trusted (as suggested by @shroffk).

Thanks anyway, I do not object to returning "invalid". If users want archiver data, they can get it without any makeover.

[jembishop]

In regards to the save restore, the operation to fetch the values should be a "bulk" operation, which fetches multiple PVs at a time, as otherwise its inefficient to make many http requests. So I don't know how suitable this api would be for that, unless there is syntax to fetch multiple PVs at once.

A prerequisite step to making the above work, is to add an interface to fetch multiple PVs in the archiver abstraction, and then to implement that for the PV archiver appliance. Somewhat confusingly, this is implemented for the other java client which is inside the epics archiver repo, but not the Phoebus client. That seems to me a good idea to tackle first.