search

singer-io / singer-tap-template

GNU Affero General Public License v3.0
68 stars 32 forks source link

Use Singer-Python Convenience Functions #8

Closed dmosorast closed 4 years ago

dmosorast commented 5 years ago

singer-python has gotten a few convenience functions that simplify more complicated pieces of writing a tap. This template should be edited to use these instead of the code currently in existence.

get_standard_metadata

This function is in the metadata module and writes things like available and table-key-properties to get fully-formed, standards compliant metadata out of the box that can be extended on a per-tap basis.

get_selected_streams Catalog has been extended with the function get_selected_streams to simplify the handling of current_stream in the state and selection metadata. All taps use this, so it should replace the function in this template.

aaronsteers commented 4 years ago

I looked into this today - was blocked/stymied by the fact that get_selected_streams() in the central library is a method of Catalog class, but the function by the same name here operates as a function and takes a dict (serialized json catalog). Any ideas?

aaronsteers commented 4 years ago

I think I've figured this out.

For ref: https://github.com/singer-io/singer-python/blob/b4b4cc25adb831dae4ec5219a4976de3fc318f77/singer/catalog.py#L150

And associated PR (just created): #10

Update: Replaced with #11 which is more holistic and resolves bugs found in the previous first pass.