search

singer-io / singer-python

Writes the Singer format from Python
https://singer.io
Apache License 2.0
537 stars 129 forks source link

Parse to UTC for datetime transform #69

Closed dmosorast closed 6 years ago

dmosorast commented 6 years ago

The transformer requires datetime strings to be pegged at UTC, or it fails. This causes datetime strings with timezones to fail, since they have a time zone attached and aren't being shifted.

This adds a function to utils that tries to parse and peg at UTC before passing into strftime and uses it within string_to_datetime in transform.py.

nick-mccoy commented 6 years ago

This also addresses this issue. We implemented a cross-platform fix similar to what was done here.

dipamacharyya commented 1 year ago

I'm running tap-onfleet --config config.json --catalog catalog.json but getting an unknown string format for my system time

Can someone help me fix this error?

INFO administrators: Starting sync INFO GET request to https://onfleet.com/api/v2/admins INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "administrators"}} CRITICAL Unknown string format: 2022-10-10 11:13:18 Pacific Daylight Time Traceback (most recent call last): File "C:\python39\lib\runpy.py", line 197, in run_module_as_main return run_code(code, main_globals, None, File "C:\python39\lib\runpy.py", line 87, in run_code exec(code, run_globals) File "C:\Python39\Scripts\tap-onfleet.exe_main.py", line 7, in File "C:\python39\lib\site-packages\singer\utils.py", line 192, in wrapped return fnc(*args, kwargs) File "C:\python39\lib\site-packages\tap_onfleet_init.py", line 91, in main sync(client, parsed_args.catalog, state) File "C:\python39\lib\site-packages\tap_onfleet_init.py", line 66, in sync counter_value = sync_stream(state, instance) File "C:\python39\lib\site-packages\tap_onfleet\sync.py", line 19, in sync_stream for (stream, record) in instance.sync(state): File "C:\python39\lib\site-packages\tap_onfleet\streams.py", line 103, in sync self.update_bookmark(state, item[self.replication_key]) File "C:\python39\lib\site-packages\tap_onfleet\streams.py", line 55, in update_bookmark if self.is_bookmark_old(state, value): File "C:\python39\lib\site-packages\tap_onfleet\streams.py", line 61, in is_bookmark_old return utils.strptime_with_tz(value) > utils.strptime_with_tz(current_bookmark) File "C:\python39\lib\site-packages\singer\utils.py", line 21, in strptime_with_tz d_object = dateutil.parser.parse(dtime) File "C:\python39\lib\site-packages\dateutil\parser_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "C:\python39\lib\site-packages\dateutil\parser_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: 2022-10-10 11:13:18 Pacific Daylight Time