Something is wrong with how we get/store versions with metadata overrides in the platform

[erohmensing]

We're writing a LOT of ADVs for salesforce: Private Zenhub Image

Those are supposed to be unique. And once it exists we shouldn't have to retrieve it anymore.

There are two issues here:

1. Fix whatever led to us even trying to create the duplicate - likely has to do with how overrides are handled on the registry side
2. Fix the unique constraint / fixup the data

Re: Number 1: this is specifically happening because we attempt to fetch the registry entry for version 2.1.1. This contains a version override for cloud which is 2.0.9 (So I guess at the time 2.1.1 was released, cloud was still pinned to 2.0.9?). This means that the cloud registry entry for 2.1.1 is actually the entry for 2.0.9 (I think? Not sure if the metadata comes from 2.0.9, or is just the metadata for 2.1.1 with the image tag 2.0.9).

Before we do that, we do a lookup for 2.1.1 (which doesn't exist in the DB), which is why we keep hitting the registry to look for it. We then persist that 2.0.9 version (which shouldnt be possible, see (2)) again and again. But our "check if version exists before grabbing and persisting" was not prepped to handle this override behavior.

While this setup is expected behavior (probably) for creating the registry itself, its definitely not what we want when retrieving registry entries for a specific version.

https://airbytehq-team.slack.com/archives/C03VDJ4FMJB/p1692296267932739

[erohmensing]

This is probably a "the registry entry should always be for the version, and when we create the actual registries we should take into account the overrides" or something of the sort

A registry entry for 2.1.1 should not have a dockerImageTag of 2.0.9

[evantahler]

Grooming:

The problem:

• source-salesfoce v2.1.1 includes the image tag/metadata for v2.0.9. It was pinned, so the registry entry took the info from the pinned version, not the metadata from this version. We shouldn't have persisted the version-pinned data to build the ADV info. Maybe this was due to an OC fix.
• The fix is needed in the platform. Overrides should override the latest entries metadata, but not the versioned metadata - that information doesn't change based on usage.
  • this could be solved by 'global' and 'per version' metadata of the connector

Fix:

• latest.json should have the pins. version.json should not, and always use the referenced version
• we need to copy the versioned file from the pinned version to the new pinned path, but don't make it latest

Sins:

• Docker image names vs image id's aren't the same
• ImageTag is the same as the connector version, but that isn't always the case

---

https://console.cloud.google.com/storage/browser/prod-airbyte-cloud-connector-metadata-service/metadata/airbyte/source-salesforce/2.1.1;tab=objects?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false

Private Zenhub Image

[erohmensing]

I believe fixing this sin would also let us undo "never upsert ADVs" which we introduced to patch this but is causing issues in the dev workflow

[erohmensing]

Things we want to support:

Happy path (no platform overrides)

• Registry: should contain registry entries of versions according to (version in latest published metadata file, whether its enabled under oss/cloud) for each connector
• LaunchDarkly overrides and OSS users using /update: Given an actor definition and version, get the corresponding registry entry for the version

Considering rollbacks:

Open question:

• dev workflow for OSS - what's a better way to handle this?

[erohmensing]

Potential to include a ref underneath oss/cloud: refs do not have to be the same as overrides

[erohmensing]

This should be closed in https://github.com/airbytehq/airbyte/pull/30699 and https://github.com/airbytehq/airbyte-platform-internal/pull/9000

Captured feedback for metadataspec v2 in this issue.

[erohmensing]

Follow up: ok i guess we can upsert ADVs