search

ControlSystemStudio / cs-studio

Control System Studio is an Eclipse-based collections of tools to monitor and operate large scale control systems, such as the ones in the accelerator community.
https://controlsystemstudio.org/
Eclipse Public License 1.0
113 stars 96 forks source link

Timezone handling in Logging #2221

Open MichaelRitzert opened 7 years ago

MichaelRitzert commented 7 years ago

I'm trying to implement logging into an elasticsearch instance based on the work done by Kunal for the alarm system (https://github.com/ControlSystemStudio/springjms2elasticsearch). I have a first version, and performance looks good. If there is interest, I can make it available somewhere.

Digging into the issue, I find that the jms2rdb version somewhat works its way around the issue of time zones by adding the time at which the message is received (i.e. a time always created at the same host) in the database as the default sort key. Sorting by CREATETIME fails in the case of users in different timezones, which we happen to have. I think there should be problems for everybody around changes from daylight saving to standard time, when times repeat (e.g. 03:00+0200 => 02:00+0100).

To tackle the underlying problem, I want to ask if it's possible to change the time format in https://github.com/ControlSystemStudio/cs-studio/blob/master/core/utility/utility-plugins/org.csstudio.logging/src/org/csstudio/logging/JMSLogMessage.java#L17 to include a "Z" at the end (or make it configurable). Existing installations should not be affected unless they rely on the exact format in own code. JMS2RDB will just pass the string through as-is, the GUI will just display it, and the sort order should not be disturbed because the change is only at the end of the string.

berryma4 commented 7 years ago

Have you looked at using Logstash as a collector? https://github.com/logstash-plugins/logstash-input-jms

shroffk commented 7 years ago

@MichaelRitzert The Timestamp format was put there when we were migrating to java8 time. We discussed this with ITER and are going to replace the hard coded format with a configurable one. Here is the PR #2219

I think the snippet in the JMSLogMessage can be replaced with final private DateTimeFormatter date_format = TimestampFormats.MILLI_FORMAT;

@kasemir is that ok?

kasemir commented 7 years ago

That JMSLogMessage format is used for communication between the alarm server (JMSAlarmMessage). So when you change it, everything related to the alarm system needs to be updated at the same time. This includes alarm server, CSS UI clients, notifier, and also web-based reporting packages.

If we do chance the format, we should change it once and then keep it fixed, i.e. update the constant in JMSLogMessage and not make it a preference.

kasemir commented 7 years ago

As for the specific time format, I agree with your approach to add something to the end, so existing logged time stamps look the same up to that added time zone detail: "yyyy-MM-dd HH:mm:ss.SSS" vs. "yyyy-MM-dd HH:mm:ss.SSS Z".

MichaelRitzert commented 7 years ago

@berryma4 Yes, I did have a look at logstash. But it does not do (out of the box) what I want it to do — accept over JMS where the messages come from, STOMP could maybe work —, and on the other hand tries to do to much processing that's not required. We do have a solution to have the IOCs log to JMS, so I don't feel like switching to another transport.

For Camel, I just wrote a tiny Processor that adds the JMS timestamp as a separate field, and that's it.

berryma4 commented 7 years ago

@MichaelRitzert, I'm also using apache camel currently. But I have a student trying the logstash jms input ... I'll report back how it goes and compare the two.