search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
94 stars 69 forks source link

bug: EmptySchemaTypeError on replication-key for entities configured #1964

Open mertbakir opened 1 year ago

mertbakir commented 1 year ago

Singer SDK Version

0.31.1

Is this a regression?

Python Version

3.10

Bug scope

Taps (catalog, state, etc.)

Operating System

Ubuntu

Description

Entities configured as FULL_TABLE replication, should not be affected by the replication-key. Meltano yaml below sets replication-key for public-table01 too, and the code crashes since public-table01 has no such column.

metadata:
      "public-*":
        replication-method: INCREMENTAL
        replication-key: update

      public-table01:
        replication-method: FULL_TABLE

Code

Traceback (most recent call last):
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 500, in invoke
    tap.sync_all()
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 459, in sync_all
    stream.sync()
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1165, in sync
    signpost = self.get_replication_key_signpost(context)
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 409, in get_replication_key_signpost
    return utc_now() if self.is_timestamp_replication_key else None
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 218, in is_timestamp_replication_key
    return is_datetime_type(type_dict)
  File "/home/ubuntu/meltano_test/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/helpers/_typing.py", line 140, in is_datetime_type
    raise EmptySchemaTypeError
edgarrmondragon commented 1 year ago

Could be fixed by a similar proposal to the one in https://github.com/meltano/sdk/issues/1198