search

willkg / crashstats-tools

Command line tools and library for interacting with Crash Stats (https://crash-stats.mozilla.org/)
Mozilla Public License 2.0
7 stars 0 forks source link

adjust reprocess to handle >10k better for automated systems #135

Open willkg opened 1 month ago

willkg commented 1 month ago

Currently, if the number of crash ids to reprocess is > 10k, the reprocess tool balks with a message like:

Reprocessing 29183 crashes sleeping 5 seconds between groups... You are trying to reprocess more than 10,000 crash reports at once. Please let us know on #crashreporting on Matrix before you do this.

That's not wildly helpful and causes automated scripts to fail. For the symbol scraper scripts which find missing symbols and then reprocess crash reports, this case comes up semi-often.

This issue covers figuring out a different behavior where we balance these things:

  1. automated systems can generally work without human intervention
  2. reprocessing requests are spread out over enough time that the processor cluster scales up and it doesn't trigger an alert

Possible ideas:

  1. If more than 10k crash ids are specified and --allow-many isn't passed, reprocess truncates the list of crash ids to 10k and only reprocesses those.
  2. If more than 10k crash ids are specified, reprocess enforces a large sleep interval between reprocess API requests or pauses X seconds every 1,000 crash ids or something like that.
willkg commented 1 month ago

I talked with Relud about this and we're going to hold off on this.