Grott Time conversion does not consider daylight savings time

[jar349]

Currently, on the east coast of the US, it is 17:40 hours. Current UTC time is 21:40. That's a 4-hour difference. In grott logs, I can see this: - Grott original time : 2023-10-15T17:40:24 adjusted UTC time for influx : 2023-10-15T22:40:24. That's a 5-hour difference. The result is that my influxdb records are timestamped one hour into the future!

Here is the code that tries to get the timezone:

try: 
    local = pytz.timezone(conf.tmzone) 
except : 
    if conf.verbose :  
        if conf.tmzone ==  "local":  print("\t - " + "Timezone local specified default timezone used")
        else : print("\t - " + "Grott unknown timezone : ",conf.tmzone,", default timezone used")
    conf.tmzone = "local"
    local = int(time.timezone/3600)

conf.tmzone is currently set to "local". That's not a timezone that pytz understands.

>>> pytz.timezone("local")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jar349/src/grott/.venv/lib/python3.11/site-packages/pytz/__init__.py", line 188, in timezone
    raise UnknownTimeZoneError(zone)
pytz.exceptions.UnknownTimeZoneError: 'local'

and so, the exception block is what runs.

>>> time.timezone
18000
>>> time.timezone/3600
5.0

That represents a 5 hour difference. But we are currently in Eastern Daylight Time (EDT) which is only a 4 hour difference. Thus, the timestamps being sent to InfluxDB2 are one hour into the future.

I suggest that in order to always get the correct timestamp in the UTC timezone, you use Python's built-in timezones like this:

# just showing what you need, I don't expect the import right here but at the top of the file
from datetime import datetime, timezone 

jsondate = "2023-10-15T17:40:24"
local_time = datetime.strptime (jsondate, "%Y-%m-%dT%H:%M:%S")
utc_time = local_time.astimezone(timezone.utc)
ifdt = utc_time.strftime("%Y-%m-%dT%H:%M:%S")

if conf.verbose :  print("\t - " + "Grott original time : ",jsondate,"adjusted UTC time for influx : ",ifdt)

this correctly shows:

- Grott original time :  2023-10-15T17:40:24 adjusted UTC time for influx :  2023-10-15T21:40:24

[anoppe]

Have you tried to change the timezone setting in the grott config? That resolved my issues regarding DST.Op 16 okt 2023 om 00:13 heeft John Ruiz @.***> het volgende geschreven: Currently, on the east coast of the US, it is 17:40 hours. Current UTC time is 21:40. That's a 4-hour difference. In grott logs, I can see this: - Grott original time : 2023-10-15T17:40:24 adjusted UTC time for influx : 2023-10-15T22:40:24. That's a 5-hour difference. The result is that my influxdb records are timestamped one hour into the future! Here is the code that tries to get the timezone: try: local = pytz.timezone(conf.tmzone) except : if conf.verbose :
if conf.tmzone == "local": print("\t - " + "Timezone local specified default timezone used") else : print("\t - " + "Grott unknown timezone : ",conf.tmzone,", default timezone used") conf.tmzone = "local" local = int(time.timezone/3600) conf.tmzone is currently set to "local". That's not a timezone that pytz understands.

pytz.timezone("local") Traceback (most recent call last): File "", line 1, in File "/home/jar349/src/grott/.venv/lib/python3.11/site-packages/pytz/init.py", line 188, in timezone raise UnknownTimeZoneError(zone) pytz.exceptions.UnknownTimeZoneError: 'local' and so, the exception block is what runs. time.timezone 18000 time.timezone/3600 5.0

That represents a 5 hour difference. But we are currently in Eastern Daylight Time (EDT) which is only a 4 hour difference. Thus, the timestamps being sent to InfluxDB2 are one hour into the future. I suggest that in order to always get the correct timestamp in the UTC timezone, you use Python's built-in timezones like this:

just showing what you need, I don't expect the import right here but at the top of the file

from datetime import datetime, timezone

jsondate = "2023-10-15T17:40:24" local_time = datetime.strptime (jsondate, "%Y-%m-%dT%H:%M:%S") utc_time = local_time.astimezone(timezone.utc) ifdt = utc_time.strftime("%Y-%m-%dT%H:%M:%S")

if conf.verbose : print("\t - " + "Grott original time : ",jsondate,"adjusted UTC time for influx : ",ifdt) this correctly shows:

• Grott original time : 2023-10-15T17:40:24 adjusted UTC time for influx : 2023-10-15T21:40:24

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

[jar349]

Have you tried to change the timezone setting in the grott config?

I don't see a timezone setting in the grott config. Can you please clarify?

Time = auto/server parameter enable/disable date/time retrieval from data record (server), default is

auto: grott decides which time is used (data record if valid otherwise Server)

If time = server Grott server time is alwas used

time = auto

☝️ is all I see and it's unclear what value I should use other than auto.

[anoppe]

I’ve got this in my config:

time = server 
timezone = Europe/Amsterdam

We’re suffering from dst here too, that’s why I investigated the timezone setting.

[jar349]

I can give that a try, although if I step back from this thread of conversation, I would observe that I shouldn't need to correctly configure anything in order for a program to successfully change an arbitrary timestamp to UTC.

[anoppe]

Correct, but then the timestamp needs to contain TZ info, so the software knows where to convert from, but the date/time from the Growatt inverter doesn't contain this info.

[johanmeijer]

Ok. Every year there seams to be some issues with DST. I tried to solve that in Grott but apparently it does not works for everybody.

I think the problem only occurs with influxdb ? Is that correct?

For witing to influx I have chosen to do this in GMT. The reason for this that is is useual to write log data in GMT (especially in time series databases) and the set timezones during the presentation of the data (e.g. Grafana). This should avoid wrong time (double records, wrong times) values etc.

I use the timezone offset in the server on which grott is running, for that or the timezone you specified in the timezone parameter in the .ini. (see also: https://github.com/johanmeijer/grott/wiki/InfluxDB-Support ).

The challenge is that Growatt inverter data records are written in the timezone you specified in the Growatt settings on the website (Growatt sends a time update configuration ones in a while). Maybe this causes the problem.

If the timezone= (e.g. timezone=Europe/Amsterdam) not works we have to find out what is going wrong. I need more information. E.g. The growatt original datarecord, the server type and setting you running on, your timezone and the settings in the grott.ini (or docker environmentals).

[johanmeijer]

I looked in the issue history and think that setting timezone= in the grott.ini should solve this issue please verify and feedback on that!

[anoppe]

@johanmeijer I can say that setting the timezone in grot.ini works for me.

[anoppe]

I think the problem only occurs with influxdb ? Is that correct?

No the same holds for HomeAssistant (i.e. mqtt data stream) in my case.

[johanmeijer]

That is strange while Grott does not change the MQTT time. Then it might be a problem of the Growatt time settings(website, phone app).

Can you sent me the information I requested above to so I can look at and test with your data?

[jar349]

I think the problem only occurs with influxdb ? Is that correct?

I only use InfluxDB2, so I cannot say for anything else.

For witing to influx I have chosen to do this in GMT. The reason for this that is is useual to write log data in GMT (especially in time series databases) and the set timezones during the presentation of the data (e.g. Grafana).

That's exactly my opinion of how to handle times.

Ok. Every year there seams to be some issues with DST. I tried to solve that in Grott but apparently it does not works for everybody.

I recommend permitting python's built-in libraries to transpose timestamps to UTC as they're likely better tested than any 3rd-party library (i.e. pytz). I think it's more likely to be correct than attempting to handle all the offset calculations yourself.

The challenge is that Growatt inverter data records are written in the timezone you specified in the Growatt settings on the website (Growatt sends a time update configuration ones in a while). Maybe this causes the problem.

Growatt may send time updates but I think it's far more likely that your users have their servers setup correctly (with regards to time) than their growatt cloud account settings.

If the timezone= (e.g. timezone=Europe/Amsterdam) not works we have to find out what is going wrong. I need more information. E.g. The growatt original datarecord, the server type and setting you running on, your timezone and the settings in the grott.ini (or docker environmentals).

All of that can be skipped if you permit python's built-in libraries to do it for you. Check out my associated PR for concrete code examples of what I mean.

[anoppe]

All of that can be skipped if you permit python's built-in libraries to do it for you. Check out my associated PR for concrete code examples of what I mean.

I have some doubts if your suggested change will work. The datetime coming from the inverter doesn't contain any timezone info, so you have to assume some tz for that incoming date time. This is where the timezone setting from the grott.ini. comes into play. Otherwise the built-in python utils will use server tz setting, which isn't necessarily the one you want to convert to (since os/docker date time settings could be misconfigured. Personally, I favor using the timezone setting from the grott.ini file together with the change you propose in your PR.

[johanmeijer]

I think using the PYTZ or Python's own method is not really relevant. It is more on what you are doing.

What I try to do is to correct the Growatt (inverter / datalogger) time to GMT. Mt first "Brilliant" idea was to take the offset of GMT from the Local server time. That did not work properly (some strange behaviour with DST indeed) so I introduced the timezone parameter in the grott.ini. This now used to calculated the GMT and works pretty well I think (at least for me).

I am aware of every programmer is doing things at his own way and how he/she is used to. At the moment I created the code PYTZ did it for me (I am not sure if I tried then Python timezone or not, maybe it was not even there yet).

For know if it works I am not going to break it (just for the coding esthetics).

[egguy]

Could I suggest to port in the future to the zoneinfo module (https://docs.python.org/3.9/library/zoneinfo.html) ? This is included in the standard library of python and this is the recommended way even by pytz.

Note

Projects using Python 3.9 or later should be using the support now included as part of the standard library, and third party packages work with it such as [tzdata](https://pypi.org/project/tzdata/). pytz offers no advantages beyond backwards compatibility with code written for earlier versions of Python.

source: https://pypi.org/project/pytz/

The only downside, it's only python 3.9+ but the official support for python 3.8 end in 11 months. To compensate this, there is backports of zoneinfo

[johanmeijer]

I will look at it in a next version.

Python version compatibility is an attention point!

[johanmeijer]

But be aware. Normally Grott does not any time processing. It uses data/time from the inverter record (which is controlled by the Growatt website/phoen app settings). Grott will not change this by default. De inverter date/time is used for MQTT and PVOutput.

Exceptions:

• When time=server is specified grott is using the server (docker container) data/time
• When time=auto and the inverter / datalogger date/time is invalid (server time is used)
• The InfluxDB date/time is being converted to GMT (UTC). When tmzone= parameter is used (eg tmzone = Europe/Amsterdam) then the GMT is being calculated as offset from this timezone. If tmzone= is not defined the server time zone is used (this is a little bit tricky I noticed). So define a tmzone = is recommended.

Be aware:

• if grottserver is used the inverter/datalogger time is set by grottserver. It uses the data Server (or docker container) data/time settings
• For docker: setting a timezone is recommended (otherwise GMT time is used).

[jar349]

My data stopped working this morning and as I investigated why, I realized that this issue has struck again. Today is the day that our clocks spring forward one hour. And I see this in the logs:

- Grott original time :  2024-03-10T04:30:02 adjusted UTC time for influx :  2024-03-10T08:30:02

However, it is currently 09:30 UTC, not 08:30 UTC.

My server knows the correct local time:

jar349@build:~/src$ date
Sun Mar 10 05:31:34 AM EDT 2024

5:30 EDT would translate correctly to 09:30 UTC. But grott uses the datetime coming from growatt instead of from the local server.

It uses data/time from the inverter record (which is controlled by the Growatt website/phoen app settings)

Previously, I mentioned that I think it is better to use the server's current time than what is in the growatt records. I still think that is the case. I don't mean to use the server's local timezone, I mean to start with the server's local time and transpose that to UTC.

[jar349]

@anoppe

I’ve got this in my config: time = server timezone = Europe/Amsterdam We’re suffering from dst here too, that’s why I investigated the timezone setting.

I've set mine to:

time = server
timezone = America/New_York

And now I'm getting the correct time. Of course, I had to restart grott to do that, so it's not clear to me that the configuration change is what solved it, or whether it was just restarting. Either way, it works now.