Open edgarrmondragon opened 2 years ago
@edgarrmondragon - I like this idea. Would we also need to be able to tackle Stream
classes like SQLStream
in which discovery can produce multiple stream instances of the same class?
What about designing discover_catalog_entries()
as a class method on the base Stream
class, adapted from SQLConnector.discover_catalog_entries()
? Normal stream classes could simply return a single entry based on dynamic or static schema
definition. And classes like SQLStream
could implement this differently, in order to return multiple (or zero) catalog entries. In the case of the SQLStream
class, the stream class could in theory call out to the SQLConnector
method we have today.
Side note: some stream types will require the config
dictionary in order to detect schema dynamically, so if we go this direction, we may need to pass the config
to the class method.
What do you think about all this?
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.
Still relevant
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.
Still relevant
Feature scope
Taps (catalog, state, stream maps, etc.)
Description
The current pattern we enforce through our documentation and cookiecutter for adding new streams to an HTTP tap is implement a custom
MyTap.discover_streams
that returns a list of the desired stream instances.This pattern may provide some consistency but it could be mildly annoying for a developer to remember they have to:
implement (once) the
discover_streams
method:A better approach might get inspiration from other OOP frameworks, like Django. There, database models/tables are defined as classes and whenever some abstraction is required for a set of models, it's also a model class, but explicitly declared as abstract, so it doesn't map to a table in the database.
So, a similar thing in the SDK might look like this:
Under the hood
MyAPIStream
would not be registered, as it's declared to be abstract,UsersStream
would be registered,discover_streams
would readUsersStream
from the underlying class registry and instantiate them appropriately.