search

mozilla-releng / buildhub2

A database index of buildhub.json files built by Mozilla
https://buildhub2.readthedocs.io/
Mozilla Public License 2.0
4 stars 13 forks source link

Backfill should use a net-mozaws-prod-delivery-firefox by default #387

Closed peterbe closed 5 years ago

peterbe commented 5 years ago

There are two buckets that we should use when using the backfill script:

  1. net-mozaws-prod-delivery-firefox
  2. net-mozaws-prod-delivery-archive

I think the region is "us-east-1" for both of these.

peterbe commented 5 years ago

Note-to-self; there are 59,591,797 in net-mozaws-prod-delivery-firefox and 400,000 keys in net-mozaws-prod-delivery-archive. That second number looks oddly rounded. Hmm...

peterbe commented 5 years ago

In list_objects_v2 you can download 1,000 keys per request. That means you have to download 59,000 of these pages. I'm currently attempting this locally on my home network and I'm up to 2,800 pages now after half an hour.

Since things are not sorted, this technique is bound to take a very very long time.

peterbe commented 5 years ago

If we do use this, it's only a matter of time till we get some sort of network outage in the middle of those 59,000 pages. According to the docs there is a "StartAfter" string. Perhaps if we dump every last key per file to disk, we can then resume from there. Needs to be tested.

peterbe commented 5 years ago

I analyzed all the keys in net-mozaws-prod-delivery-archive by printing about 1% randomly. The keys are things like this:

1% random keys in net-mozaws-prod-delivery-archive pub/mobile/nightly/2010/03/2010-03-20-01-mobile-lorentz/en-US/deb_name.txt pub/mobile/nightly/2010/09/2010-09-13-02-mozilla-central-maemo5-qt/en-US/fennec-2.0b1pre.en-US.linux-gnueabi-arm.tar.bz2 pub/mobile/nightly/2010/12/2010-12-31-05-places-maemo5-qt/fennec_4.0%7Eb4%7E20101231041550_armel.deb pub/mobile/nightly/2011/04/2011-04-08-03-mozilla-2.1-maemo5-qt/en-US/deb_name.txt pub/mobile/nightly/2011/06/2011-06-15-04-tracemonkey-android/tracemonkey_tegra_android_test-remote-tdhtml-build739.txt.gz pub/mobile/nightly/2011/07/2011-07-27-08-18-00-mozilla-central-android/mozilla-central_tegra_android_test-mochitest-1-build1782.txt.gz pub/mobile/nightly/2011/09/2011-09-13-04-02-57-jaegermonkey-android/jaegermonkey_tegra_android_test-reftest-1-build113.txt.gz pub/mobile/nightly/2011/10/2011-10-30-03-11-01-mozilla-central-android/mozilla-central_tegra_android_test-jsreftest-2-build1452.txt.gz pub/mobile/nightly/2011/12/2011-12-10-04-20-32-mozilla-aurora-macosx/jsshell-mac.zip pub/mobile/nightly/2012/01/2012-01-16-19-27-02-oak-macosx/fennec-12.0a1.en-US.mac.txt pub/mobile/nightly/2012/02/2012-02-15-03-11-55-mozilla-central-android-xul/mozilla-central_tegra_android-xul_test-remote-ts-build165.txt.gz pub/mobile/nightly/2012/03/2012-03-13-04-02-42-maple-android/maple_tegra_android_test-remote-ts-build29.txt.gz pub/mobile/nightly/2012/04/2012-04-05-10-52-00-mozilla-central-android-xul/mozilla-central_tegra_android-xul_test-mochitest-8-bm19-tests1-tegra-build403.txt.gz pub/mobile/nightly/2012/04/2012-04-29-04-20-06-mozilla-aurora-android/mozilla-aurora_tegra_android_test-crashtest-2-bm19-tests1-tegra-build284.txt.gz pub/mobile/nightly/2012/05/2012-05-23-09-34-09-oak-android-xul/oak_tegra_android-xul_test-mochitest-2-bm19-tests1-tegra-build10.txt.gz pub/mobile/nightly/2012/06/2012-06-08-04-20-08-mozilla-aurora-android/mozilla-aurora_tegra_android_test-remote-trobopan-bm20-tests1-tegra-build143.txt.gz pub/mobile/nightly/2012/06/2012-06-30-03-05-32-mozilla-central-android-xul/mozilla-central_tegra_android-xul_test-jsreftest-2-bm19-tests1-tegra-build696.txt.gz pub/mobile/nightly/2012/07/2012-07-29-03-05-16-mozilla-central-android/en-US/gecko-unsigned-unaligned.apk pub/mobile/nightly/2012/09/2012-09-05-04-02-03-fx-team-android/fx-team_tegra_android_test-mochitest-1-bm20-tests1-tegra-build361.txt.gz pub/mobile/nightly/2012/10/2012-10-17-03-05-48-mozilla-central-android-noion/mozilla-central_tegra_android-noion_test-mochitest-3-bm10-tests1-tegra-build12.txt.gz pub/mobile/nightly/2012/11/2012-11-20-04-02-04-oak-android/oak_tegra_android_test-reftest-3-bm22-tests1-tegra-build13.txt.gz pub/mobile/nightly/2012/12/2012-12-25-03-07-57-mozilla-central-android-armv6/mozilla-central-android-armv6-nightly-bm49-build1-build22.txt.gz pub/mobile/nightly/2013/01/2013-01-25-04-02-01-elm-android-armv6/elm_tegra_android-armv6_test-mochitest-8-bm10-tests1-tegra-build15.txt.gz pub/mobile/nightly/2013/02/2013-02-22-04-20-21-mozilla-aurora-android/fennec-21.0a2.multi.android-arm.json pub/mobile/nightly/2013/03/2013-03-18-04-20-13-mozilla-aurora-android-x86/en-US/mar pub/mobile/nightly/2013/04/2013-04-11-03-09-25-mozilla-central-android/mozilla-central_panda_android_test-crashtest-2-bm44-tests1-panda-build301.txt.gz pub/mobile/nightly/2013/05/2013-05-06-00-40-15-mozilla-aurora-android/mozilla-aurora_panda_android_test-robocop-1-bm45-tests1-panda-build13.txt.gz pub/mobile/nightly/2013/05/2013-05-28-00-40-18-mozilla-aurora-android-armv6/fennec-23.0a2.multi.android-arm-armv6.checksums pub/mobile/nightly/2013/06/2013-06-24-00-40-20-mozilla-aurora-android/mozilla-aurora_panda_android_test-mochitest-5-bm44-tests1-panda-build134.txt.gz pub/mobile/nightly/2013/07/2013-07-19-03-02-04-mozilla-central-android/mozilla-central_panda_android_test-mochitest-5-bm44-tests1-panda-build336.txt.gz pub/mobile/nightly/2013/08/2013-08-10-03-02-06-mozilla-central-android/mozilla-central_tegra_android_test-remote-tp4m_nochrome-bm22-tests1-tegra-build536.txt.gz pub/mobile/nightly/2013/09/2013-09-01-03-02-18-mozilla-central-android-armv6/en-US/fennec-26.0a1.en-US.android-arm-armv6.checksums.asc pub/mobile/nightly/2013/09/2013-09-21-04-02-01-fig-android/fig_panda_android_test-jsreftest-2-bm29-tests1-panda-build5.txt.gz pub/mobile/nightly/2013/10/2013-10-12-00-40-04-mozilla-aurora-android/en-US/mbsdiff pub/mobile/nightly/2013/10/2013-10-31-00-40-03-mozilla-aurora-android/mozilla-aurora_tegra_android_test-plain-reftest-2-bm19-tests1-tegra-build444.txt.gz pub/mobile/nightly/2013/11/2013-11-20-04-02-03-oak-android/oak_panda_android_test-robocop-2-bm42-tests1-panda-build8.txt.gz pub/mobile/nightly/2013/12/2013-12-07-03-02-03-mozilla-central-android/mozilla-central_panda_android_test-mochitest-gl-bm42-tests1-panda-build102.txt.gz pub/mobile/nightly/2013/12/2013-12-24-03-02-03-mozilla-central-android-x86/mbsdiff pub/mobile/nightly/2014/01/2014-01-15-03-02-29-mozilla-central-android-armv6/mozilla-central_tegra_android-armv6_test-mochitest-7-bm88-tests1-tegra-build123.txt.gz pub/mobile/nightly/2014/01/2014-01-29-04-02-27-holly-android/holly_tegra_android_test-mochitest-7-bm99-tests1-tegra-build22.txt.gz

Note that they appear to be roughly in order and the last one is Jan 2014.