Unix Time support for MicroPython

[amotl]

Hi there,

within #51, we are discussing CayenneLPP Unix Time type compatibility with MicroPython.

@smlng said:

This adds dependencies to datetime and tz. I am not sure about appropriate support in MicroPython - though the status of MicroPython is a bit unclear to me, too.

@amotl said:

Indeed, we are using the library on MicroPython through our Terkin Datalogger and would like it to stay compatible with MicroPython.

I will check whether the current implementation would work using the MicroPython datetime module [1] and document the process within this issue.

With kind regards, Andreas.

[1] https://github.com/micropython/micropython-lib/blob/master/datetime/datetime.py

[amotl]

Problem

Pycom MicroPython 1.20.2.rc6-0.10.2-vanilla-squirrel-nosmartconfig [v1.20.1.r2-122-gd82a6f43e-dirty] on 2020-03-06; FiPy with ESP32
Type "help()" for more information.
>>>
>>> from cayennelpp import LppFrame
>>> frame = LppFrame()
>>> frame.add_unix_time(0, 1590010843)

>>> print(frame)
LppFrame(data = [
  LppData(channel = 0, type = Unix Time, value = (1590010843,))
])

>>> frame.bytes()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cayennelpp/lpp_frame.py", line 61, in bytes
  File "cayennelpp/lpp_data.py", line 61, in bytes
  File "cayennelpp/lpp_type.py", line 221, in lpp_unix_time_to_bytes
  File "datetime.py", line 1919, in <module>
  File "datetime.py", line 1836, in _create
AttributeError: type object 'tzinfo' has no attribute '__new__'

It does not work out of the box yet. Bummer!

Mitigation

After following https://github.com/micropython/micropython-lib/issues/319 and applying https://github.com/micropython/micropython-lib/pull/338 to the MicroPython datetime module, at least importing tzinfo works now.

from datetime import tzinfo

[amotl]

Problem

However, now I am getting:

>>> from cayennelpp import LppFrame
>>> frame = LppFrame()
>>> frame.add_unix_time(0, 1590010843)
>>> frame.bytes()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cayennelpp/lpp_frame.py", line 61, in bytes
  File "cayennelpp/lpp_data.py", line 61, in bytes
  File "cayennelpp/lpp_type.py", line 231, in lpp_unix_time_to_bytes
  File "datetime.py", line 1370, in fromtimestamp
TypeError: can't convert float to int

Indeed, CPython supports fractions of seconds on the time.gmtime() method:

>>> import time
>>> time.gmtime(1590010843.0)
time.struct_time(tm_year=2020, tm_mon=5, tm_mday=20, tm_hour=21, tm_min=40, tm_sec=43, tm_wday=2, tm_yday=141, tm_isdst=0)

while MicroPython (at least Pycom MicroPython) doesn't:

>>> import time
>>> time.gmtime(1590010843.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't convert float to int

It only accepts Integers here:

>>> time.gmtime(1590010843)
(2020, 5, 20, 21, 40, 43, 2, 141)

Mitigation

This has been fixed like

# MicroPython does not accept fractions of seconds to "time.gmtime()".
t = int(t)

[amotl]

Problem

This is next:

>>> from cayennelpp import LppFrame
>>> frame = LppFrame()
>>> frame.add_unix_time(0, 1590010843)
>>> frame.bytes()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cayennelpp/lpp_frame.py", line 61, in bytes
  File "cayennelpp/lpp_data.py", line 61, in bytes
  File "cayennelpp/lpp_type.py", line 231, in lpp_unix_time_to_bytes
  File "datetime.py", line 1374, in fromtimestamp
ValueError: need more than 8 values to unpack

This is because MicroPython returns an 8-tuple, omitting the last slot tm_isdst.

Mitigation

# CPython returns a 9-tuple, while MicroPython return an 8-tuple,
# omitting the last "is_dst" slot.
try:
    y, m, d, hh, mm, ss, weekday, jday, dst = converter(t)
except ValueError:
    y, m, d, hh, mm, ss, weekday, jday = converter(t)

[amotl]

Success

Encoding works now:

>>> from cayennelpp import LppFrame
>>> frame = LppFrame()
>>> frame.add_unix_time(0, 1590010843)
>>> frame.bytes()
bytearray(b'\x00\x85^\xc5\xa3\xdb')

Decoding also does:

>>> print(LppFrame.from_bytes(b'\x00\x85^\xc5\xa3\xdb'))
LppFrame(data = [
  LppData(channel = 0, type = Unix Time, value = (datetime.datetime(2020, 5, 20, 21, 41, 20, tzinfo=datetime.timezone.utc),))
])

[amotl]

So, we conclude it should be safe to merge #51, when taking into account that the MicroPython datetime module needs some updates, which have been implemented within datetime.py through https://github.com/daq-tools/pycopy-lib/commit/5c8e8342.

[amotl]

Sebastian suggested to

use the same code as for the generic uint but with the correct type for unix timestamp and use integer as input and output. The user would then do something like

# Encode
from datetime import datetime
frame = LppFrame()
frame.add_unix_time(datetime.now().timestamp())

Decode

dt = datetime.fromtimestamp(frame.data[0].value)

[smlng]

Let's keep this open until we have this feature in - independent of the implemented solution.

[smlng]

basic implementation done in #56