search

mozilla-releng / buildhub2

A database index of buildhub.json files built by Mozilla
https://buildhub2.readthedocs.io/
Mozilla Public License 2.0
4 stars 13 forks source link

No Nightly build update after 20230915094248 #1042

Closed pascalchevrel closed 11 months ago

pascalchevrel commented 11 months ago

The tool doesnt list any new Firefox nightly after BuildID 20230915094248

gbrownmozilla commented 11 months ago

I checked Sentry: there are a couple of issues, but they seem low-frequency and about the same before / after 2023/09/15.

gbrownmozilla commented 11 months ago

I also checked the buildhub.json files on a.m.o. -- they look okay to me ... but I could be missing something.

releng is trying to get access to logs in https://mozilla-hub.atlassian.net/browse/SVCSE-1465 ; until then, I can't think of anything else to check.

@smarnach can you see what's going wrong?

bhearsum commented 11 months ago

This appears to have been caused by https://github.com/mozilla-releng/scriptworker-scripts/pull/840, which I believe was finally deployed to production on Friday.

Buildhub is configured to listen on an SQS queue, which is populated by S3 events in AWS. Disabling the S3 uploads means it never learned of any of the new files.

bhearsum commented 11 months ago

https://github.com/mozilla-releng/scriptworker-scripts/pull/846 should let us pick up new builds when deployed. We'll need to take other steps to backfill. In theory, buildhub supports backfilling, but I don't know if it's viable to run in practice; there doesn't seem to be support for just pointing it at certain files or directories.

gbrownmozilla commented 11 months ago

I verified that new nightlies were picked up by buildhub after https://github.com/mozilla-releng/scriptworker-scripts/pull/846 .

gbrownmozilla commented 11 months ago

I manually backfilled thunderbird and firefox nightlies, by copying missing *buildhub.json files from archive.m.o. to s3.

Working on missing candidates...

gbrownmozilla commented 11 months ago

Firefox 118.0 has been backfilled now; that should be the only one missing from firefox/candidates/.

gbrownmozilla commented 11 months ago

Thunderbird 118.0b5 build1/build2 have been backfilled now.

That should catch us up on all the missing data.

smarnach commented 11 months ago

Let me know if you want me to run the backfill command in production. I mean, we could just try? Even if it takes three days, at least we'd get the data back with little work.

gbrownmozilla commented 11 months ago

Thanks Sven, but let's leave it as it is: I think we are in good shape now.