support for milliseconds in timestamp attributes (Datetime type)

[agaldemas]

Hello, As soon as Orion should be used in smart industry, even it's not aimed at controlling industrial process, it could be valuable to support milliseconds in timestamp attributes of type DateTime, for the moment the milliseconds in a DateTime attribute are removed even specified in the injected value like :

"TimeInstant": {
    "type": "DateTime", 
    "value": " 2019-01-25T11:11:11.111Z"}
} 

the milliseconds part

.111

are not stored in mongo (set to 0):

"TimeInstant" : {
            "type" : "DateTime",
            "creDate" : 1548499006,
            "modDate" : 1548499006,
            "value" : 1548414671.0,
            "mdNames" : []
        }

and then can't be restore in query results to Orion:

    "TimeInstant": {
        "type": "DateTime",
        "value": "2019-01-25T11:11:11.00Z",
        "metadata": {}
    }

I tried several other format, without success, unless using a structured value, I didn't tried in metadata. we can relate with: #432 and from #2670, not asking microseconds (CERN have already their own tools for LHC ;O), I learnt that already milliseconds are used internally in orion for logs. Even milliseconds are not mentioned in ISO 8601, at the digital & blockchain era we live, it is becoming important to keep this information in entities timestamp data... NGSI V2 evolution ? In NGSI-LD spec: https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.01.01_60/gs_CIM009v010101p.pdf it is envisaged to use YYYY-MM-DDThh:mm:ss,ssssssZ format

(microseconds are there but why ',' instead of '.' ? worth to be signaled that '.' is more generally used ?)

I made different tests using NGSI V1 and V2 orion and json iot agent latest docker images:

• using NGSI V1 for data injection , it is still possible to get milliseconds (type is ISO8601) in notifications, but only in metadata in the database
• but using injection in NGSI V2 milliseconds are lost (whatever type is DateTime or ISO8601)

So, sooner or later milliseconds/microseconds support will be needed respect to NGSI specification. Thanks to think about this suggestion to manipulate by default DateTime with milliseconds, Or at least for TimeInstant in a first step, to be consistent with NGSI V1 behavior in V2

[agaldemas]

Wonder why no one even post a comment on this issue ? Do you want I rewrite it in a simpler way ? Thanks in advance for your feeback

[fgalan]

I saw the issue but I'm afraid I haven't yet have time to analyze and provide a answer. I'll do as soon as possible. In the meanwhile maybe some other member of the Orion community could provide feedback on it.

[agaldemas]

Hi @fgalan, thanks for your feedback, to do it simple: I noticed that milliseconds are lost at the insertion in mongo when using DateTime or ISO8601 types with V2 requests, with V1 requests and ISO8601 type for timestamp milliseconds are kept but in metadata only, as a formatted date. And globally milliseconds are not handled for dateModified and dateCreated (modDate & creDate in mongo).

[fgalan]

(Finally, I have been able to have some time to have a look and anwser :)

You are right: precision higher than seconds (i.e. miliseconds, microseconds, etc.) are not supported in DateType NGSIv2 attribute type. That's the reason of issue #2670 (in some sense #432 can be seen as the implementation detail of #2670).

With regards to NGSIv1, it's working because DateTime has no meaning in that API version. By the way, we strongly encourage not using NGSIv1, as it is deprecated. You can achieve the same effect in NGSIv2 using something like this:

"TimeInstant": {
    "type": "DateTimeWithMilis", 
    "value": " 2019-01-25T11:11:11.111Z"
}

However, note that in this case CB interpretes the attribute as a text string. Features related with DateTime (such as date-based filter like GET/v2/entities?q=TimeInstant<2019-01-01, support for timezones, etc.) are not available. But depending on your use case it could suffice. (By the way, this is a similar case of using dates in structured values and metadata: they aren't DateTime but text).

That is just a workaround. Of course, the best option will be to address #2670 (and #432). If somebody volunteer for that, I'll be more than happy to discuss about implementation options and provide hints and code review along the process :) Thanks!

[fgalan]

With regards to YYYY-MM-DDThh:mm:ss,ssssssZ, it's the first time I see a format like that :) In my experience, not sure if an standard or not, but the dot (.) is typically used as separator.

Maybe @jmcanterafonseca , which knows NGSI-LD much better than me, can provide additional feedback.

[agaldemas]

Hello @fgalan, @jmcanterafonseca About NGSI LD and timestamp format, I made a feedback to ETSI, with a proposal to change in the next version of specification to use '.' instead of ',' as separator, and to accept also milliseconds as well. May be I can transfer you my email, so that you can also support my suggestion !

[fgalan]

What ISO8061 tells about fractions (http://www.loc.gov/standards/datetime/iso-tc154-wg5_n0038_iso_wd_8601-1_2016-02-16.pdf section "4.2.2.4 Representations with decimal fraction") is:

the decimal fraction shall be divided from the integer part by the decimal sign specified in ISO31-0, i.e. the comma [,] or full stop [.]. Of these, the comma is the preferred sign

From my point of view, NGSI-LD should use the same.

Having said that, if you think that only "." should be supported (i.e "," forbidden) and you want to propose that to ETSI you can also have my support :)

[agaldemas]

You're right @fgalan, If you don't mind, I'll send a new email to ETSI, copy to you, to refer iso 8601, and accept both [,] & [.] for second decimal fraction.

[fgalan]

If you don't mind, I'll send a new email to ETSI, copy to you, to refer iso 8601, and accept both [,] & [.] for second decimal fraction.

Ok!

[tstorek]

Hi, we have the use case we would like to monitor and control smart energy systems over Orion in combination with IoT-Agents. For thermal systems of accuracy with down to one second might seem enough. However, when it comes to electrical systems we need to measure more frequently with higher sampling rate. Performancewise this should be possible, however, using quantumleap to monitor the system it would be great if orion would support also milliseconds. The Workaround mentioned above does not seem to be applicable forever. As I also understand the roadmap in developments related to the energy sector these applications are highly desirable .

Hence, is there any progress on this issue or the related ones?

[fgalan]

I think the situation doesn't have changed since I tool the following (some comments above):

Of course, the best option will be to address #2670 (and #432). If somebody volunteer for that, I'll be more than happy to discuss about implementation options and provide hints and code review along the process :) Thanks!

By the way, @agaldemas, regarding:

If you don't mind, I'll send a new email to ETSI, copy to you, to refer iso 8601, and accept both [,] & [.] for second decimal fraction.

Did you get any answer from ETSI? It would be very weird they don't adhere NGSI-LD to a higher level standard as ISO8601 is...

[fgalan]

Related question at SOF (which answer should be update when this issue progresses):

https://stackoverflow.com/questions/62516873/when-iot-agent-sends-data-within-milliseconds-the-values-are-not-notified-to-dr/62661681#62661681

[fgalan]

PR #3673 (which is the sum of PRs #3668 and #3672) implements milliseconds functionality for DateTime attributes and metadata into master.

@agaldemas @tstorek it would be great if you coult test it (using :latest from dockerhub for the Orion container) and provide feedback, please.

[tstorek]

@fgalan Just tested :) apparently it works fine. THANK YOU VERY MUCH! when will you have the next release. In production I do prefer stable versions instead of latest, especially, because once changed you cannot downgrade anymore. It seems timestamp was an integer previously.

[fgalan]

Thank you for the report and for kinds word! :)

We don't have an exact date for the release of 2.5.0 but probably it will be done when the log improvements get finished (PR https://github.com/telefonicaid/fiware-orion/pull/3696). Maybe some weeks.

[fgalan]

@tstorek Orion 2.5.0 has been released some minutes ago ;)

[fgalan]

Milliseconds support has been implemented for Orion 2.5.0 and, except if somebody tells otherwise, it works as expected. Closing issue.