search

willkg / crashstats-tools

Command line tools and library for interacting with Crash Stats (https://crash-stats.mozilla.org/)
Mozilla Public License 2.0
8 stars 0 forks source link

add download stats and estimated completion to fetch-data #149

Open willkg opened 2 days ago

willkg commented 2 days ago

If you're using fetch-data to download data for multiple crash reports, it'd be nice for it to output download stats and an estimated completion that takes 429s and retrying and all that into account.

Right now it output something like this:

> supersearch --num=10 | fetch-data --processed --no-raw --no-dumps --no-overwrite crashdata
No API token provided. Set CRASHSTATS_API_TOKEN in the environment.
Skipping dumps and protected data.
8a88187c-8150-48f0-990c-8923e0240918: fetching processed crash
eff16bb6-cf17-4afd-ae39-e9b0b0240918: fetching processed crash
236473fb-fc67-43ea-8568-65e1c0240918: fetching processed crash
53a11671-9bd8-47a0-952f-095fe0240918: fetching processed crash
9bf480e0-597a-42eb-b104-0620d0240918: fetching processed crash
dbb1f4a2-bb5a-4ec1-bff6-2f4830240918: fetching processed crash
32cbe67d-5e7a-4fde-a3c8-16abe0240918: fetching processed crash
2dad8839-79f7-46ab-bb79-f80670240918: fetching processed crash
5b3a5efd-4e16-4751-badd-a04350240918: fetching processed crash
9d4c01ed-5340-4dc9-8e43-b05920240918: fetching processed crash
willkg commented 2 days ago

After the change in PR #150, this does:

$ supersearch --num=105 | fetch-data --processed --no-raw --no-dumps --overwrite --workers=10 --stats crashdata
Using API token: xxx
Using 10 workers.
Downloaded (0/105) 0:00:52.308396
Downloaded (100/105) 0:00:00.514721
Completed in 0:00:08.

That's a little goofy since it prints an estimate after the first one, but good enough for a first pass.