Decouple connector development from Airbyte monorepo

[sherifnada]

Tell us about the problem you're trying to solve

Today connectors generated via the connector development process are created inside the Airbyte repository. It is not well documented or supported how a developer can decouple this code from the Airbyte repo, taking into account not only which repo the code lives in, but especially CI/CD. All of this must be setup from scratch in a custom way by the connector developer. This inhibits the adoption of custom connectors via Airbyte, as the monorepo becomes a bottleneck.

Describe the solution you’d like

I want the connector generator:

• to create a file at a directory of my choosing on my local machine.
• I should be able to immediately convert this to a standalone (private or public) github repo
• The repo should come preconfigured with CI/CD configurations that work out of the box
• As a nice bonus, airbyte cloud customers should also have their docker images pushed to a private, Airbyte managed dockerhub repository which they can use out of the box to use their Airbyte connector on Airbyte Cloud

[noahkawasakigoogle]

+100!

Other ideas I've thought of, related:

• Maybe have a publicly hosted set up script (mirrors checked in generate.sh file?), like dockers to remove the need to clone airbyte's main repo at all. Theoretically, an Airbyte connector builder shouldn't have to touch any airbyte core stuff at all, if the development process allows for that.
• On CI/CD - Airbyte supplied Github Actions, CircleCI, maybe other big CI platforms too. Github Actions could even support being packaged as a custom airbyte-connector-test action. CircleCI might support something similar.
• Airbyte also includes what is needed to build the connector image, like gradle build files. If Airbyte wants to commit to gradle forever, it might be worth having an official airbyte gradle plugin that takes all the custom airbyte python/other language stuff out of the repo. All of that build logic could be usable outside of airbytehq. Depends on whether people will prefer to use non gradle build systems for their connectors though.
• Airbyte test tools are usable outside of airbytehq repo. For example, SAT packaged as a python library pip install airbyte-connector-source-acceptance-tests. Same deal for destinations, etc.
• Alternative idea from code/directory/template based generator, create a separate airbyte-connector-template repo and have people clone it, rename it, set an alternate remote to their own github acccount, and use pre-existing templates. Another way to remove the need to clone airbyte's main repo.

Probably lots of build dependency stuff from airbyte core that needs to be reworked to support extracting so much stuff out of the main repo for this vision.

[sherifnada]

Yup agreed on a lot of the functionality proposed here. We can probably do these things separately (improve the generator is separate from splitting it out of the airbyte repo, though one could copy a file/directory directly out of the GitHub monorepo instead of clone the whole repo). I would love to find a path that doesn't require airbyte devs to work across two/more repos if possible.

[girarda]

To give a sense of urgency to this ticket: cloning the repo now takes more than one minute, which is not a good first step for first-time connector developers.

[justinzamm]

Stumbled into this while doing my research. Can someone guide me and let me know if the following is possible at the moment please.

• Generate a source connector using the code generator, without cloning the whole repository. For example we do not need all the connectors (airbyte-integrations/connectors) to be pulled and everything else that is not relevant to the generator.

   cd airbyte-integrations/connector-templates/generator
  ./generate.sh

• For generate.sh bypass the interactive helper application and pass the template and connector name as arguments directly.

For some context I am building a script which will automate the generation of a connector, copies a manifest file and pushes the connector as a Docker image to a repository.