Builder: Adding a custom source to PyAirbyte using the no-code builder

aaronsteers commented 6 months ago

Summary

This blog post would focus on how to build a custom connector manifest using the Low-Code builder and then executing that connector from PyAirbyte.

Definition of Done

Blog post illustrating how to build custom connectors and then executing them from PyAirbyte.

Resources to Assist

PyAirbyte tutorials: https://github.com/airbytehq/quickstarts/tree/main/pyairbyte_notebooks
PyAirbyte Documentaton: https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started
PyAirbyte API Reference: https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started#api-reference
Sample blog post: https://airbyte.com/blog/manage-and-orchestrate-airbyte-connections-using-python

New feature PR (WIP) to further streamline this:

https://github.com/airbytehq/PyAirbyte/pull/175

fvgm-spec commented 5 months ago

Hi @aaronsteers , I am interested on participating writing this piece, I have recently written a tutorial in Airbyte blog.

aaronsteers commented 5 months ago

Hi @aaronsteers , I am interested on participating writing this piece, I have recently written a tutorial in Airbyte blog.

Great! Assigning you to this issue. 👍

Let us know if you have any questions or issues!

fvgm-spec commented 5 months ago

Hi @aaronsteers, do you agree with @avirajsingh7? is it worth writing two tutorials on the same topic?

bindipankhudi commented 5 months ago

Good point. let's wait for @aaronsteers for feedback here.

aaronsteers commented 5 months ago

Hi, @fvgm-spec. Apologies for the confusion. This particular item was to focus on no-code connectors, which can be represented and "installed" by copying yaml files.

The related item:

https://github.com/airbytehq/PyAirbyte/issues/247

was intended to focus on custom Python connectors, which require a pip install.

I think both make sense - and I've update the related item to more clearly focus on the nature of Python connectors specifically.

aaronsteers commented 5 months ago

@fvgm-spec - If you do continue with this post, there is one WIP PR which you should be aware of:

https://github.com/airbytehq/PyAirbyte/pull/175

That has come about organically and you can leverage that PR's branch instead of main for a more streamlined experience.

(Added this PR link above as supporting resource.)

fvgm-spec commented 5 months ago

Hi @aaronsteers thanks for both updates, I will be catching the post on building no-code connectors maybe tomorrow, and I will take a look into the PR.

Regards.

aaronsteers commented 5 months ago

@fvgm-spec - https://github.com/airbytehq/PyAirbyte/pull/175 has shipped!

The fresh-out-of-the-oven v0.12.0 release of PyAirbyte includes a new source_manifest argument in the experimental get_source() implementation (which also includes a Docker executor).

API Docs here: https://airbytehq.github.io/PyAirbyte/airbyte/experimental.html#get_source

And PR that shipped this feature is here:

https://github.com/airbytehq/PyAirbyte/pull/175

Please let us know if you have questions or if you run into any issues.

aaronsteers commented 5 months ago

In case it is helpful, I wanted to share this Loom demo video that I recorded this morning. The link to my code is in the video description.

https://www.loom.com/share/a80d3bcb42444a069c0ef03087044e99?sid=b987bebe-c5ed-48d6-8cd5-2ee19985ee3f

fvgm-spec commented 5 months ago

Great @aaronsteers thanks for such a great video, it explains a lot on what needs to be covered in the blog post.

fvgm-spec commented 5 months ago

Hi @aaronsteers I have made public the post intended to explain how to build custom connectors in Airbyte Connector Builder, and then use it in data pipelines with PyAirbyte.

Please let me know the further steps, or if I can continue with another issue from the list. Regards.

aaronsteers commented 5 months ago

Hi, @fvgm-spec! I appreciate your patience in the review. I had a chance to review and the content seems easy to follow and informative. All the technical details also look correct. I also appreciate the attachment of notebook to help users reproduce the results.

@bindipankhudi and @marcosmarxm - I'll let you take final review review and begin steps to publish. From my side, the content and technical implementation part looks very good. 👍

bindipankhudi commented 5 months ago

Sounds good! I will queue it for publishing @aaronsteers. @fvgm-spec could you share a brief bio for yourself, so we can add that to the blog post.

fvgm-spec commented 5 months ago

Hi @bindipankhudi and @aaronsteers thanks for the review, my bio will be "Felix is a Data Engineer with experience in multiple roles related to Data. He's building his professional identity being an adaptable professional who can navigate constantly changing technological environments."

bindipankhudi commented 5 months ago

Awesome! Thank you @fvgm-spec. Will keep you posted as we publish this (in a day or two).

bindipankhudi commented 5 months ago

@fvgm-spec , your blog has been published! - https://airbyte.com/blog/adding-a-custom-source-to-pyairbyte-using-the-no-code-builder 🚀 , it's also accessible in Top stories on Airbyte blogs (on the right side). If you would like to publish it elsewhere, please wait for one week. Thank you! - fyi @marcosmarxm

airbytehq / PyAirbyte-Hackathon