feature: display progress bar when downloading multi-page responses

[hydrosquall]

Motivation

For a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered

We know how many pages there are, so it may be possible to indicate how many remain.

https://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367

Resources

• Using the existing Click API:
  • https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars
• Loading spinner: https://github.com/pavdmyt/yaspin
• Progress bar: https://github.com/tqdm/tqdm

[simonw]

That's interesting - yeah it looks like the number of pages can be derived from the Link header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.

https://docs.github.com/en/rest/guides/traversing-with-pagination