search

airbytehq / airbyte-python-cdk

The Python CDK empowers hundreds of Airbyte connectors, including low-code and no-code connectors.
https://airbytehq.github.io/airbyte-python-cdk/
MIT License
0 stars 3 forks source link

Ability to test manifest-only connectors directly from the CDK repo #64

Open aaronsteers opened 1 week ago

aaronsteers commented 1 week ago

In absence of a --use-local-cdk option for manifest-only connectors, we can't test them against dev branches in this repo.

We'll either need to update airbyte-ci to accept --use-local-cdk for manifest-only connectors, or we would need some other test flow to confirm downstream connectors are not breaking (or conversely to confirm their bugs are fixed) by changes here in this repo.

aaronsteers commented 1 week ago

@alafanechere, @natikgadzhi, @ChristoGrab - Logging for visibility.

My best suggestion is that we add something like--use-local-cdk for manifest-only connectors, but another option would be adding --use-cdk-ref support.

  1. --use-local-cdk - In this former case, we'd run a manifest-only connector from an adjecently-cloned CDK repo.
  2. --use-cdk-ref - In this case, we'd point the test to a specific commit.

My rationale for preferring "option 1": while option 1 can subsume the requirements of "option 2" simply by cloning both and switching to the correct branches (we're already doing this for Python connectors), option 2 does not work for not-yet-committed changes. Which means: a developer making changes to the CDK could not test their changes are successful until/unless they commit and push their changes. If we add option 1, we can still always follow up with option 2 - since they are not mutually exclusive.

Maybe there is an "option 3" I'm not aware of (@alafanechere to confirm?), but I think we otherwise need to prioritize an "option 1" solution highly (24-48 hours?) even if a perfect options isn't yet available.

@alafanechere - does option 1 sound sensible to you as well? And if so, do you have any capacity to help us deliver this? I think this is in the "asap" category of post-migration action items. If you are swamped I'll try to pick this up tomorrow (Tuesday).