dogsheep / github-to-sqlite

Save data from GitHub to a SQLite database
https://github-to-sqlite.dogsheep.net/
Apache License 2.0
405 stars 43 forks source link

Turn GitHub API errors into exceptions #21

Closed simonw closed 4 years ago

simonw commented 4 years ago

This would have really helped in debugging the mess in #13. Running with this auth.json is a useful demo:

{"github_personal_token": ""}
simonw commented 4 years ago

I'm going to do this now to help figure out the latest error in #13:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite", line 11, in <module>
    load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()
  File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 237, in commits
    utils.save_commits(db, commits, repo_full["id"])
  File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 345, in save_commits
    for commit in commits:
  File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 207, in fetch_commits
    if stop_when(commit):
  File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 224, in stop_when
    db["commits"].get(commit["sha"])
TypeError: string indices must be integers
simonw commented 4 years ago

I'm happy with this pattern: https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18