Unable to run app.get() for some objects in a knack form due to datetime issue

[jenna-jordan]

When running app.get("object_1") on a knack form, I get an error that prevents me from getting any app data. Full traceback is copied below. The knack application id is 5cc35f4acaf774317c5c74b4 (see the metadata here) . I'm currently on version 1.1.1 of knackpy. This error only occurs for this one knack form thus far (all others tested have been successful and have not encountered this error), but does occur for some objects (object_1, object_19) in the form, but does not for others (object_4).

app.get("object_1")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\app.py", line 246, in get
    self.records[container_key] = self._records(container_key, generate)
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\app.py", line 281, in _records
    for record in data
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\app.py", line 281, in <listcomp>
    for record in data
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\record.py", line 35, in __init__
    self.raw = self._handle_record()
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\record.py", line 152, in _handle_record
    record = self._correct_knack_timestamp(record, self.timezone)
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\record.py", line 197, in _correct_knack_timestamp
    val["unix_timestamp"], timezone
  File "c:\Users\167753\Documents\GitHub\civis_pipelines\knack-env\lib\site-packages\knackpy\utils.py", line 64, in correct_knack_timestamp
    dt_utc = datetime.datetime.fromtimestamp(timestamp, tz=datetime.timezone.utc)
OSError: [Errno 22] Invalid argument

I suspect that this error is caused by some malformed/mistyped dates. I've included one such date from the downloaded JSON file below as an example:

"field_25":"07/03/0198","field_25_raw":{"date":"07/03/0198","date_formatted":"07/03/0198","hours":"12","minutes":"00","am_pm":"AM","unix_timestamp":-55903046400000,"iso_timestamp":"0198-07-03T00:00:00.000Z","timestamp":"07/03/0198 12:00 am","time":720}

Obviously we want to correct these dates, however, this error is preventing me from fetching any data from the app at all. So, it would be best if this error could be handled in such a way that app data can still be accessed/processed.

[johnclary]

@jenna-jordan I just ran a quick test of _correct_knack_timestamp

import datetime
import pytz

timezone = pytz.timezone("America/New_York")
mills_timestamp = -55903046400000
timestamp = mills_timestamp / 1000
dt_utc = datetime.datetime.fromtimestamp(timestamp, tz=datetime.timezone.utc)
dt_naive = dt_utc.replace(tzinfo=None)
dt_local = timezone.localize(dt_naive)

No issue, the result looks fine:

datetime.datetime(198, 7, 3, 0, 0, tzinfo=<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>)

Reading the datetime docs, this might be a platform-specific issue. Are you running this on Windows?

if the timestamp is out of the range of values supported by the platform C localtime() function, and OSError on localtime() failure. It’s common for this to be restricted to years from 1970 through 2038.

Here's an SO post with the cross-platform way to handle this. TIL. I'd estimate this as a 1 to patch it. Will be backwards compatible.

I believe we're going to talk internally next week about how our team is going to prioritize these fixes going forward. In the meantime feel free to submit a PR @jenna-jordan.

cc @mateoclarke @frankhereford

[jenna-jordan]

When I was running this in the debugger, I was on a Windows computer. When we run workflows in production, they are run in a docker container running Ubuntu. I will test out to see if the problem occurs using Ubuntu as well.

[jenna-jordan]

Confirmed that it runs fine in the docker container (Ubuntu) - seems the issue is only with Windows. cc @johnclary