Closed bnchrch closed 1 year ago
This goes for any place in which we don't already have an actor definition - the /update
endpoints where OSS users can choose any version they like also apply here
If we move a connector on [oss|cloud] from v1 to v2 with a 6 month deprecation window, how do we ensure we freeze v1s metadata related settings in time
If everything lives on the ADV and not the actor definition, the breaking changes phase 0 should handle this correctly. What is currently not handling this correctly is the fact that we upsert the ADVs. There are 2 issues here
If we add a connector + version to [oss|cloud] that the platform has never seen, how do we populate its metadata related values?
If we do this via override or /update endpoint, I think we need to request this info from the metadata service.
Grooming:
ADV Properties in Question:
Options:
v1.2.3
, New Tag -> {dockerTag: "v1.2.3", releaseStage: "generally_available"}
- now in LD you provide JSONoss_registry-GIT_BRANCH
)TODO:
--> SOMEONE WRITE A SMALL TECH SPEC PICKING WHAT WE SHOULD DO
https://github.com/airbytehq/airbyte/issues/27077 for not updating ADV part
In my mind, the "entry point" for a connector version from our registry should be its metadata - and the docker image just happens to be how we run that connector. So, to have a version override that says "v1.2.3" means go grab everything for this version (not just a "docker image override").
In the platform today when processing overrides we go to the spec cache bucket to grab the spec for the given version. I think this should be replaced with looking at our registry and pulling all the version-specific information from our versioned metadata.
If this were an API, I would imagine something like /connectors/source-faker/versions/1.2.3 and getting that info. With the current model of serving versioned metadata in files, this is sort of already available in YAML form at https://connectors.airbyte.com/files/metadata/airbyte/source-faker/2.1.0/metadata.yaml - though what's a bit weird is this isn't the exact contents that would have been returned in the registry served at https://connectors.airbyte.com/files/registries/v0/cloud_registry.json. There's some processing that happens and we fill in the spec and I'm not sure what else.
In any case, I don't think this is a full registry we need (i.e. a list) or any seeding that we have to do platform side. I think what we need is more a way to request a "registry entry" for a specific verison on demand.
The logic on the platform should follow the same as what happens for spec today, just filling in more information. As a refresher, when an override is found in LaunchDarkly, we:
So, we should replace step 2i with something that provides the rest of the fields (the metadata) and the rest should just work.
I have a similar worldview as the above! It would make the platform rely on our service a lot more, but I think that's probably an inevitability?
We should do the same if we get an OSS /update
to a version that doesn't exist already in the DB (e.g. prereleases) - the user would have to be online to grab the prerelease metadata, but they'd have to be so in order to get the docker image too. If they can hit the api when they pull the docker image, we could find a way to make that work
Grooming notes:
Grooming:
Problem
For any OLD connector version, how do we get the registry entry that was in the catalog but no longer is?
Solution
Lets update our registry generator to cache connector entries
For example if you are looking for the last registry entry for destination-bigquery:1.2.19 on cloud you would go to
https://connectors.airbyte.com/files/metadata/airbyte/destination-bigquery/1.2.19/cloud.json
OLD CONVERSATION BELOW
The Current System
source-s3:1.2.3
and cloud can havesource-s3:2.0.0
source-s3:eds-prerelease-123
Questions for the team
The 6th point is the awkward turtle and we need to answer the question(s):
Related Conversations
Normalization Overrides: https://airbytehq-team.slack.com/archives/C03VDJ4FMJB/p1685552842556829 Rollbacks: https://airbytehq-team.slack.com/archives/C03VDJ4FMJB/p1685567791734259