dogsheep / github-to-sqlite

Save data from GitHub to a SQLite database
https://github-to-sqlite.dogsheep.net/
Apache License 2.0
405 stars 43 forks source link

Exception running first command: IndexError: list index out of range #16

Closed jayvdb closed 4 years ago

jayvdb commented 4 years ago

Exception running first command without an existing db or auth.

> mkdir ~/.github/coala
> /usr/bin/github-to-sqlite repos ~/.github/coala coala
Traceback (most recent call last):
  File "/usr/bin/github-to-sqlite", line 11, in <module>
    load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')()
  File "/usr/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py", line 163, in repos
    utils.save_repo(db, repo)
  File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 120, in save_repo
    to_save["owner"] = save_user(db, to_save["owner"])
  File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 61, in save_user
    return db["users"].upsert(to_save, pk="id", alter=True).last_pk
  File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1135, in upsert
    extracts=extracts,
  File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1162, in upsert_all
    upsert=True,
  File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1105, in insert_all
    row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
IndexError: list index out of range
jayvdb commented 4 years ago

I re-tried after doing auth, and I get the same result.

simonw commented 4 years ago

I'm pretty sure this means I need to upgrade the underlying sqlite-utils dependency. I saw the same bug in twitter-to-sqlite.

jayvdb commented 4 years ago

I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.

I would also need tokenless and netrc support, but I would be happy to add those bits.

simonw commented 4 years ago

This should have been fixed by #20 and #23

@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.

Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project!