search

BuzzCutNorman / tap-mssql

Singer Tap for MS SQL built with Meltano Singer SDK.
MIT License
2 stars 9 forks source link

Batch Message creation errors out when datetime present #2

Closed BuzzCutNorman closed 2 years ago

BuzzCutNorman commented 2 years ago

When you extract a time that has a datetime column the batch message dump fails because it cannot deal with this data type.

Error

TypeError: Object of type datetime is not JSON serializable

Error from meltano invoke tap-mssql

PS C:\development\projects\test-mssql> meltano invoke tap-mssql            
2022-10-25T19:31:18.298852Z [info     ] Environment 'dev' is active
2022-10-25 12:31:20,504 Beginning full_table sync of 'dbo-PostLinks'...
2022-10-25 12:31:20,504 Tap has custom mapper. Using 1 provided map(s).
{"type": "SCHEMA", "stream": "dbo-PostLinks", "schema": {"properties": {"Id": {"type": ["integer"]}, "CreationDate": {"format": "date-time", "type": ["string", "null"]}, "PostId": {"type": ["integer", "null"]}, "RelatedPostId": {"type": ["integer", "null"]}, "LinkTypeId": {"type": ["integer", "null"]}}, "type": "object", "required": ["Id"]}, "key_properties": ["Id"]}
2022-10-25 12:31:20,644 INFO METRIC: {"metric_type": "timer", "metric": "sync_duration", "value": 0.14062070846557617, "tags": {"stream": "dbo-PostLinks", "context": {}, "status": "failed"}}
2022-10-25 12:31:20,644 INFO METRIC: {"metric_type": "counter", "metric": "record_count", "value": 999, "tags": {"stream": "dbo-PostLinks", "context": {}}}
2022-10-25 12:31:20,644 INFO METRIC: {"metric_type": "counter", "metric": "batch_count", "value": 0, "tags": {"context": null, "stream": "dbo-PostLinks"}}
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\Scripts\tap-mssql.exe\__main__.py", line 7, in <module>
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\singer_sdk\tap_base.py", line 506, in cli
    tap.sync_all()
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\singer_sdk\tap_base.py", line 379, in sync_all
    stream.sync()
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\singer_sdk\streams\core.py", line 1083, in sync
    self._sync_batches(batch_config, context=context)
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\singer_sdk\streams\core.py", line 1051, in _sync_batches
    for encoding, manifest in self.get_batches(batch_config, context):
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\singer_sdk\streams\core.py", line 1240, in get_batches
    gz.writelines(
  File "C:\development\projects\test-mssql\.meltano\extractors\tap-mssql\venv\lib\site-packages\singer_sdk\streams\core.py", line 1241, in <genexpr>
    (json.dumps(record) + "\n").encode() for record in chunk
  File "C:\Program Files\Python39\lib\json\__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "C:\Program Files\Python39\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Program Files\Python39\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Program Files\Python39\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable
BuzzCutNorman commented 2 years ago

I started at the function to_jsonschema_type but found it skips this all together

BuzzCutNorman commented 2 years ago

I think need to extended the json.JSONEncoder as explained in this article https://dev.to/kfuquay/extending-pythons-json-encoder-7k0