Closed sarfata closed 5 months ago
We should have a way to specify the resampling per period length
See also #89 History/time series api and #363 Track API.
Are you sure this is actually much less verbose? Looking at it with a human eye is misleading as we pretty print and compare. When you remove whitespace and CR/LF the effect is different.
Waaay back I did a size comparison of delta vs sparse full format, there was very little difference, mostly around how much of the path was common to the values. Compression gives us a huge size reduction because of the repetition.
But we do need a format for history.
BTW we should avoid calling the playback and snapshot functionality 'history'. Its becoming confusing since they are likely to be different api's and formats. eg
1) 'Playback' - extension to /signalk/v1/stream
to replay data
2) 'Snapshot' - extension to /signalk/v1/api
to retrieve data at a point in time
3) 'History' - new api to get bulk historic data by date-range.
- what samples the client requests: min/max/average/first/last and a way to have for example min and max for navigation.speedOverGround for each time period
I do recognize that this is useful and I have just tried a few formats that feel very contrived. I think the best way might be to use the source for this.
For example:
{
"path": "navigation.speedOverGround",
"source": { "label": "aggregation.max", type: "max", "originalSource": { "label": "NMEA1" } },
"values": [ 12.2, 12.1, 10.9 ]
}
I think this fits well into our general model. Thoughts?
Notes:
period length / time slice, in query and in response
Yes for the query.
For the response, it would be great to know the time slice so you can directly access time t with values[(t - startTime) / period]
but that means we really enforce the fact that all the timestamps are precisely following the period. Are we ready to include that requirement? Maybe it's an optional field but when it's provided timestamp n
must be equal to startTime + n * period
?
If we decide to do this, I will extend the validation to actually verify this too (also need to check that they are all in chronological order and that they are not repeated).
Are you sure this is actually much less verbose? Looking at it with a human eye is misleading as we pretty print and compare. When you remove whitespace and CR/LF the effect is different.
Waaay back I did a size comparison of delta vs sparse full format, there was very little difference, mostly around how much of the path was common to the values. Compression gives us a huge size reduction because of the repetition.
So I added support for 'history' format (very WIP) to my conversion tool in strongly-signalk
and did some tests using some real log files people have sent me during charted sails development.
File | Original size | Original size + GZ | Size in SK Delta | Size in SK Delta + GZ | Size in SK History | Size in SK History + GZ |
---|---|---|---|---|---|---|
Velocitek logfile (.vcc which is GPX like) (9 hours) | 1.7 MB | 208k | 3.4 MB | 312kB | 1.3MB | 236 kB |
Log from Cassiopeia (1 hour SignalK log with 176k updates / 217k values) | 29 MB | 3.2 MB | 33 MB | 1.5 MB | 28 MB | 1.1 MB |
Log from tkurki in SK (38mn SignalK log with 37k updates / 65k values) | 7.1 MB | 916 kB | 8.1 MB | 372kB | 5.8 MB | 280 kB |
Expedition san-francisco.csv example 16k updates/92k values | 1.4 MB | 328kB | 8.1MB | 632kB | 2.7MB | 316kB |
Conclusions:
Would be interesting too to measure the size 'in memory' of the file in different engines. I know how to do this with Chrome but not with NodeJS. If anyone has ideas, let me know!
In memory size will quickly overrun the little RPi :-(
The history implementation should be using streaming, as we can easily produce arbitrarily large datasets. That doesnt mean ws etc, just streaming internally so memory stays tiny.
If we reply with a gzipped stream of updates (via http) then the format problem is also solved. Size is excellent, and we already have the handling code.
Unless we agree on the use-cases, I think we can all be right at the same time and still disagree on the best solution. Reading comments here, I think we have very different scenarios in mind. We need to clarify what use-cases we are trying to solve for so we can make a decision: What are the types of apps that are consuming this history format? What are they doing with it?
My main use-case is displaying how data series change over time on a map or a graph. To do this, I need all the data in memory at once and I need to be able to quickly find the min/max of the value (for the scale of a graph for example) and quickly access data in an index format. This is very expensive to do with the delta format and that is why I would be happy to see a different format (such as the one I proposed in this PR) that would be much easier to consume directly.
I agree with @sarfata, we should start from real world use cases.
We can also first build some of the applications, at the risk of doing things a few times over, and then come back to the spec issue once the most predrominant use cases have been worked out.
We had a lengthy discussion with Rob over Slack, where the main point (from my point of view at least) was that to enable efficient stream-based handling the format should have self contained units: for example having first a list of timestamps and then the corresponding data forces the consumer to collate data from different parts of the dataset. Instead putting the related items together allows more efficient processing in most circumstances.
Too bad GeoJSON does not allow extending the coordinates...
My additional use case is to collect track, depth and other info within a bounding box and display on the chart. Same for wind, sog, etc for polar comparisons, engine performance comparisons, etc Also data export to the cloud with intermittent connection. @tkurki since you need the full message to make the array in ram work, you could just ingest the stream and build the array locally?
To me the whole point of having a history API is to provide fast and convenient access to historical data with different aggregation options.
Digesting the original delta stream does not fit any of those criteria.
Cross-posting the format being discussed:
[
{"2015-03-07T12:37:10.523+13:00":[
{"vessels.urn:mrn:imo:mmsi:234567890":[
{ "navigation.position": {
"$source": "a.suitable.path",
"average":{
"longitude": 24.9025173,
"latitude": 60.039317
}
}
},
"environment.depth.belowSurface": {
"$source": "a.suitable.path",
"avg":2.5,
"max": 2.8,
"min": 2.5
}
]
},
...more vessels
]
},
...more timestamps
]
I'm open to other formats that: 1) are 'packetised' so we can write/read them with low resources and they are resilient when partially sent/recieved over intermittent links. 2) Handle multi-vessels 3) Handle complex combinations of paths, nulls, missing data, etc
A thought here: a lot of the discussion is about the use case, and what suits it. Obviously thats different for different use cases. I think we should concentrate a the best format to transfer data, not to process data.
This format could be used to dump data for backup, transfer data between signalk instances, consolidate data to a different timeslice, or return data for a history query. It should handle bad connections, potentially huge data sizes, and multi-vessel/complex query responses. In the delta format we only considered now(), so it was keyed on the vessel. In this case the natural key is timestamp
, hence the natural self-contained unit or packet is one timeSlice
The production of data, and the clients use of the data is actually implementation detail. There are so many use-cases that we cant optimise a generic format for any specific one. If we really need to do that, then we should have a specific API for that use-case. aka /track/
It sounds like you are after a transfer format and we should create it separate from a more use case driven format.
I agree with @tkurki , seems like we should have a separate API for the transfer use cases.
So, how this format will looks like? Like @rob42 mentioned above?
@sarfata @rob42 @tkurki this seems stale. In any case there are conflicts. Please update or we should close and revisit at a later time. Thoughts?
I think this is still useful. While the PR is stale now, the issue is going to come up again as soon as we use history in more complex ways. It also relates to #543 since both need a high volume and very efficient transfer format
Closing as stale, to be implemented as OpenApi description in the future, see https://github.com/SignalK/signalk-server/pull/1653
Summary
Propose to create a new SignalK data format that can be used to convey data about multiple SignalK objects (vessels, aton, aircraft, etc) over a period of time.
Motivation
Multiple developers are working on history related features and there is a strong interest in agreeing on one common data format. Discussion was started and continues on Slack #history-api.
In this proposal, I suggest a new format for the following reasons:
update
object andvalue
object for each timestamp represented. It also repeats the key names for each timestamp;This format can be used as both a "log" file format and a format served over HTTP from a server.
Detailed design
History data format
The history object provides essential information about the data included:
self
if it is known (it is not required because in some cases we do not know)generator
key to include information on who generated this file (especially useful for logfiles)And then a list of
objects
. Each object is identified by acontext
like we already do ondelta
objects but then for each we provide:A few important notes:
null
s.timestamps
array must be equal to the length of theproperties.*.values
arrays.null
in thevalues
array.History REST endpoint
(WIP - Need more work on this)
Drawback
This is one more format.
Alternatives
See the slack #history-api archive or https://docs.google.com/document/d/1s4_lHVVyKJlfacpq5LcUPQEZHvU5nSVtdtE0BxSbbSw/edit# for a summary of other proposals discussed.
Using delta format
Using geo-json
navigation.position
key is not included