Update time_index generation policy

[taliaga]

As regards method _get_time_index,

With higher priority than dateModified it should look for the timestamp in the metadata (https://github.com/Fiware/dataModels/pull/302/files#diff-9308877b8fc4a861aa6d718690d52f82R331) and the TimeInstant element, in the case of https://github.com/telefonicaid/iotagent-node-lib#the-timeinstant-element

[chicco785]

(y)

[agaldemas]

Hello, We need this feature to be able to store device timestamped data, coming from IOT agents, through TimeInstant element, may I/you open a new PR, where I can fix this behavior: Look for & give priority to TimeInstant over dateModified could be a good title ? Nevertheless by experience it's better to keep any timestamp information we can grab for analytics, so I suggest to keep and store dateModified from Orion notification

[chicco785]

Hi, This is something we have been discussing: how to specify a different attribute metadata as TimeIndex. We have been thinking of adding some metadata to the notification to handle that.

A PR is welcome, of course. Let’s open a google doc to collaborative design the solution, what do you think?

Cheers, Federico

Il giorno 26 gen 2019, alle ore 10:34, agaldemas notifications@github.com ha scritto:

Hello, We need this feature to be able to store device timestamped data, coming from IOT agents, through TimeInstant element, may I/you open a new PR, where I can fix this behavior: Look for & give priority to TimeInstant over dateModified could be a good title ? Nevertheless by experience it's better to keep any timestamp information we can grab for analytics, so I suggest to keep and store dateModified from Orion notification

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[agaldemas]

Sorry, for late reply, I went through other issues with orion ngsi behavior regarding milliseconds in timestamp (DateTime type)... I wanted to be sure that we could provide an efficient way to retrieve milliseconds from Orion notification, given an update context sent by an iot agent, and TimeInstant attribute, since the aim is to be able to handle device attributes timestamp at milliseconds/microseconds resolution. After some testing, I realized there is no way for the moment except using metadata, confirmed by @fgalan in an issue discussion on the subject: support for milliseconds... Think we should invite him in the discussion ? What do you think ?

[chicco785]

@c0c0n3 is landing shortly a pr to configure using headers the attribute to be used as timeindex. (Which means TimeInstant will be supported). This could be a starting point, but it will not cover metadata (simply because as of today, ql does not store metadata). TimeIndex is (if I recall correctly) stored as epoch timestamp. Supporting microseconds may be not possible, it depends on what crate.io allows.

[taliaga]

That's right, the database stores time_index as a long representing milliseconds since epoch. Support for microseconds could still be implemented in QL but it'd require a big effort to compensate for that. Out of curiosity, what's the scenario you're working on requiring microseconds accuracy?

[agaldemas]

Hello For the moment only milliseconds should be ok, but I am afraid lits orly in metadata let me confirm it asap Cu

Le lun. 11 févr. 2019 à 08:35, Tomas notifications@github.com a écrit :

That's right, the database stores time_index as a long representing milliseconds since epoch. Support for microseconds could still be implemented in QL but it'd require a big effort to compensate for that. Out of curiosity, what's the scenario you're working on requiring microseconds accuracy?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/smartsdk/ngsi-timeseries-api/issues/93#issuecomment-462238988, or mute the thread https://github.com/notifications/unsubscribe-auth/ATWJozlbIr9FCb-M0L8qhOvWjzm_gTygks5vMR0_gaJpZM4XwFyB .

[agaldemas]

To summarize my tests about milliseconds: if using NGSI V1 from IOT agent json to Orion: TimeInstant is transmitted like this in notification sent by Orion:

"TimeInstant": {
    "type": "ISO8601", 
    "value": " 2019-01-25T11:11:11.111Z"}
}

if using NGSI V2 TimeInstant is transmitted like this in notification sent by Orion:

"TimeInstant": {
    "type": "DateTime", 
    "value": " 2019-01-25T11:11:11.00Z"}
'metadata': {'dateCreated': {'type': 'DateTime', 'value': '2019-01-25T11:11:11.00Z'}
}

So the milliseconds are lost by Orion with NGSI V2 between IOT agent and Orion which means:

1/ We have to use another attribute name than TimeInstant, and a specific type different from "DateTime" and "ISO8601", ==> @fgalan suggested to use "type": "DateTimeWithMilis" ("DateTimeMs" should be OK), hence it won't be taken by Orion as a timestamp.

2/ We need to parse the TimeInstant value because it's not an epoch timestamp, to build time_instant for CrateDB.

[c0c0n3]

@taliaga, @chicco785, @agaldemas see #161.

[fgalan]

Some extra information about this:

1/ We have to use another attribute name than TimeInstant, and a specific type different from "DateTime" and "ISO8601", ==> @fgalan suggested to use "type": "DateTimeWithMilis" ("DateTimeMs" should be OK), hence it won't be taken by Orion as a timestamp.

Just to make clear that I suggested that (details in issue: https://github.com/telefonicaid/fiware-orion/issues/3412#issuecomment-461109330) as a temporal workaround. Of course, the good solution will be address that issue and properly support miliseconds in Orion an any other component able to interact with it (as I understand QuantumLeap is). However, I'm not sure about the complexity of that modification in Orion code (it needs to be analyzed, taking into account aspects such as backward compatibility).

[taliaga]

Closed by https://github.com/smartsdk/ngsi-timeseries-api/pull/169