DaedalusG
commented
1 year ago A manual run of the drift check ignoring the version tag check shows no drift -- this will be verified but indicates that the drift isn't real and the UI is indicating the wrong drift to to an inaccurate entry in the versions table
[ec2-user@ip-172-31-55-120 ~]$ helm upgrade --install --set "migrator.args={drift,--db=frontend,--version=v5.1.4,--skip-version-check}" sourcegraph-migrator sourcegraph/sourcegraph-migrator --version 5.1.4
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
Release "sourcegraph-migrator" has been upgraded. Happy Helming!
NAME: sourcegraph-migrator
LAST DEPLOYED: Fri Jul 21 11:14:49 2023
NAMESPACE: default
STATUS: deployed
REVISION: 9
TEST SUITE: None
[ec2-user@ip-172-31-55-120 ~]$ k get jobs
NAME COMPLETIONS DURATION AGE
migrator-nxavh 1/1 4s 6s
[ec2-user@ip-172-31-55-120 ~]$ k logs job/migrator-nxavh
✱ Sourcegraph migrator 5.1.4
ℹ️ Connection DSNs used: frontend => postgres://sg:password@pgsql:5432/sg
Attempting connection to postgres://sg:password@pgsql:5432/sg...
✅ Connection to "postgres://sg:password@pgsql:5432/sg" succeeded
ℹ️ Locating schema description
ℹ️ Reading schema definition in Local file (/schema-descriptions/v5.1.4-internal_database_schema.json)... Schema not found (open /schema-descriptions/v5.1.4-internal_database_schema.json: no such file or directory). Will attempt a fallback source.
✅ Schema found in GitHub (https://raw.githubusercontent.com/sourcegraph/sourcegraph/v5.1.4/internal/database/schema.json).
✅ No drift detectedInferring that migrations were run correctly but the versions table wasn't upgraded correctly.
Migrator fails on upgrade
Issue reported in v4.4
During an AMI upgrade, if an AMI instance is upgraded via the standard upgrade procedure, some drift may be introduce if the instance is not rebooted as instructed in step 10.
Reproduction
An instance s initialized in v4.4
Upgrade to v4.5.1 - reboot
Observe failing migrator pods after startup and attachment of old volume
reboot the EC2 machine and check for drift
Upgrade to v5.0.6
Upgrade to v5.0.6
Upgrade to v5.1.4
Version
Drift in UI
Manual drift check
Database version
Reboot in version 5.1.4
Checking drift against the v5.0.6 version
Actual logs omitted, but this drift output is the same as is registered in the Update page
Summary
On upgrade from v5.0.6 to v5.1.x the migrator isn't correctly initializing and setting the db state to the correct version. It is however likely running schema migrations. Either the migrations are being applied correctly and the schema drift in the updates page is the result of a bad
versionstable entry. Or the schema migrations aren't being run by theupcommand.Given these conditions once the direction issue of the
upoperations failure is identified this can likely be solved manually by correct use of theupgradecommand. A hypothesis as to the root cause of this issue is the tagging of a5.0.6image set insourcegraph/deployrepo. Migrators image definitions insourcegraph/sourcegraphrepo may not handle correctly for the extra/missing version.