airbytehq / terraform-provider-airbyte

Terraform Provider for Airbyte API
https://api.airbyte.com/
MIT License
47 stars 21 forks source link

Support custom connectors #36

Open aballiet opened 1 year ago

aballiet commented 1 year ago

A lot of Airbyte users implement custom connectors using python CDK or low-code CDK.

Currently it's not possible to configure sources leveraging custom connector.

maver1ck commented 11 months ago

It will be nice to have source_generic and destination_generic resources.

aballiet commented 11 months ago

In the end I made a Terraform provider for Airbyte OSS version which supports custom connectors.

You can find a full example here if interested 👍

lideke commented 8 months ago

@aballiet looks nice your work. I'll try to use it we were waiting so long for this. Thanks

aballiet commented 8 months ago

@aballiet looks nice your work. I'll try to use it we were waiting so long for this. Thanks

Yes it's frustrating...

Tell me if you experience any issue 😊

lideke commented 8 months ago

@aballiet looks nice your work. I'll try to use it we were waiting so long for this. Thanks

Yes it's frustrating...

Tell me if you experience any issue 😊

i'll keep you update wondering if I can add custom destination with your repo ? Because we are using airbyte as reverse ETL and airbyte is not ready for this so we keep it only as custom destination

aballiet commented 8 months ago

@aballiet looks nice your work. I'll try to use it we were waiting so long for this. Thanks

Yes it's frustrating... Tell me if you experience any issue 😊

i'll keep you update wondering if I can add custom destination with your repo ? Because we are using airbyte as reverse ETL and airbyte is not ready for this so we keep it only as custom destination

I think you can using this : https://registry.terraform.io/providers/aballiet/airbyte-oss/latest/docs/resources/airbyte_destination_definition

lideke commented 8 months ago

@aballiet nice work

stevenmurphy12 commented 8 months ago

@aballiet +1 nice work. I'm in the early phases on testing your provider and it's looking promising.

It's unfortunate the official provider doesn't support this yet, Airbyte recently added the ability to create custom connectors in the Airbyte API - https://reference.airbyte.com/reference/createsource

NumberPiOso commented 8 months ago

@aballiet +1 amazing work. Is this compatible with sources created using Connector Builder Intro or it's YAML or this only targeted for connectors created using the CDK.

In a more philosophical question, would you want your code to be part of the official terraform-airbyte provider? or is there any reason you prefer not to.

NumberPiOso commented 8 months ago

@aballiet +1 nice work. I'm in the early phases on testing your provider and it's looking promising.

It's unfortunate the official provider doesn't support this yet, Airbyte recently added the ability to create custom connectors in the Airbyte API - https://reference.airbyte.com/reference/createsource

@stevenmurphy12 this only allows to create a source from an already defined connector in the instance, isn't it? Allows for creation of source but not the underlying connector for that source.

stevenmurphy12 commented 8 months ago

@NumberPiOso - Yes that's correct.

Being able to create the actual connector definition was not a particularly high priority for me, though I was able to use @aballiet provider to do that - https://registry.terraform.io/providers/aballiet/airbyte-oss/latest/docs/resources/airbyte_source_definition.

I'm currently trying out a hybrid. Using the airbyte-oss provider for:

And using airbytehq/airbyte for:

aballiet commented 8 months ago

@aballiet +1 amazing work. Is this compatible with sources created using Connector Builder Intro or it's YAML or this only targeted for connectors created using the CDK.

In a more philosophical question, would you want your code to be part of the official terraform-airbyte provider? or is there any reason you prefer not to.

It's working for both. We build connector using low-code (YAML) and build docker image with it. See here for details, it will generate you a Dockerfile when selecting on the following :

image

The option ❯ Configuration Based Source - Generate a Source that is described using a low code configuration file.

Then you can use your YAML manifest to build the connector, push it to your artifact registry and use the Terraform provider with it 👍

yannik207 commented 3 months ago

Does anybody know wether there is an official solution for a source definition?