search

airbytehq / PyAirbyte

PyAirbyte brings the power of Airbyte to every Python developer.
https://docs.airbyte.com/pyairbyte
Other
221 stars 35 forks source link

Feature Request: Ability to load custom connectors built using no code builder #107

Open bindipankhudi opened 7 months ago

bindipankhudi commented 7 months ago

To be explored: We might have to point to a connector python package or load a yaml file.

natikgadzhi commented 7 months ago

@aaronsteers I took a quick look! airbyte-cdk has sources.declarative.ManifestDeclarativeSource which loads a yaml file and provides an Airbyte CDK Source implementation on top of it.

It's used in connector builder CDK wrapper here.

The easiest way to add this to PyAirbyte would be:

  1. Extract an abstract class interface from pyairbyte.Source
  2. Add a PyAirbyte DeclarativeSource that implements PyAirbyte.Source, but internally wraps the Declarative Source from the CDK instead of managing pip dependency and shelling out into it.

I think I have everything I'd need to hack on it and make a nice tutorial, but time. 🙃

ifokeev commented 6 months ago

Need this feature a lot. Airbyte has a very nice no-code builder, but launching jobs from airbyte itself is very slow. Pyairbyte could be a good replacement

aaronsteers commented 6 months ago

Related work: