Open trevjonez opened 4 months ago
I may try deploying against the latest rolling docker tag to see if that unblocks me.
But it seems the release process is potentially flawed in allowing things to go out without being in lockstep on critical shared code so I wanted to raise the question directly.
rolling version published to docker hub may 8th seems to be working and things seems to be flowing thru the system correctly again.
I am also seeing this behavior on my end. @trevjonez, what exactly did you end up pinning to?
I am also seeing this behavior on my end. @trevjonez, what exactly did you end up pinning to?
I used the rolling tag from the may 8th build. Pulled and retagged to suit my needs
Interesting. If I use 24.5.1
for everything and the latest rolling
for the worker, I am getting the same issue.
Did you verify all your migrations got ran? I've had to do it manually from the api container every time I update. This round it had issues and I had to edit the migration table and run it again to get it to go thru then finally undo the edit. There was a ton of trial and error over about an hour so I can't say exactly the steps I took.
I believe it was some error about an out of order dependent migration.
I do see some odd messages popping from the api
:
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration user_measurements.0001_initial is applied before its dependency codecov_auth.0054_update_owners_column_defaults on database 'default'.
I am a little hesitant to start messing with migrations - 🤞 someone from the team can chime in 😬 .
yeah that was the one i hit as well. I believe what I did was renamed the thing to 0001_initial_temp. Then ran the migrations again to get a new error (something like can't modify the user-measurements because it had actually already ran the edit). put it back as 0001_initial then ran again.
What worked for me:
django_migrations
table (user_measurements
)('user_measurements', '0001_initial');
What worked for me:
- Delete latest migration entry in
django_migrations
table (user_measurements
)Re-run migrations
- This should introduce a few new entries and then say something like "user_measurements already exists")
- Re-add original errored entry at the end
('user_measurements', '0001_initial');
The image docker.io/codecov/self-hosted-worker:24.5
uses a version of codecov shared
that seem to contain a commit before this change https://github.com/codecov/shared/pull/205/commits/d86d4667805723264205bda88f39f4a3ced2f45c
The worker is crashing when it runs an SQL query with a field that does not exist, though the migration (with the same name) for it has run. I'm assuming the migration with the same name was run by self-hosted-api, but with the correct migration content.
Looking at the worker image I notice the field name rollout_identifer
instead of rollout_universe
in the migration
docker run -it --entrypoint cat docker.io/codecov/self-hosted-worker:24.5.1 /usr/local/lib/python3.12/site-packages/shared/django_apps/rollouts/migrations/0005_featureflag_is_active_featureflag_platform_and_more.py
requirements.in contains a reference to the right commit for the shared lib though:
$ docker run -it --entrypoint cat docker.io/codecov/self-hosted-worker:24.5.1 /worker/requirements.in
https://github.com/codecov/shared/archive/148b7ae3a6d4cdfc554ba9ca8b911c13e82d77b8.tar.gz#egg=shared
<snip>
Is this a packaging issue in your image build?
Pinning at codecov/self-hosted-worker:rolling@sha256:e287e7e4557fdb1d79dbee1849aa2e84ee442d48b778df6b28ae16a5dcc94d48
seem to work well for now
After updating my selfhosted instance and struggling thru getting migrations to run I am left with the following errors:
After digging around for a bit I found this commit that renamed the column: https://github.com/codecov/shared/commit/d86d4667805723264205bda88f39f4a3ced2f45c#diff-552087a68d49f71285c998a00138656ec63de61039c6bbac698e935c4d9d40d1
Then looking at my own DB tables I am seeing the migration was ran.
But it seems that possibly the 2024.5.1 docker images for worker and api were built using different versions of
shared
? (worker repo isn't git tagged so I can really only guess what commit the image was built from) https://github.com/codecov/worker/blob/5c7c8927010514c2e53f1b14728a57de9be3d102/requirements.txt#L375https://github.com/codecov/codecov-api/blob/self-hosted-24.5.1/requirements.txt#L405