search

dogsheep / github-to-sqlite

Save data from GitHub to a SQLite database
https://github-to-sqlite.dogsheep.net/
Apache License 2.0
405 stars 43 forks source link

stargazers command, refs #4 #8

Closed simonw closed 4 years ago

simonw commented 4 years ago

Needs tests. Refs #4.

simonw commented 4 years ago

Also need to take #5 into account - if this command creates incomplete user records, how do we repair them?

And make sure that if we run this command first any future commands that populate users don't break (probably just a case of using alter=True in a few places).

simonw commented 4 years ago

Adding the view from #10 would be useful here too.

simonw commented 4 years ago

Spotted a tricky problem: running github-to-sqlite starred stargazers.db results in an incomplete simonw record. It creates a proper record for me thanks to this bit:

https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/cli.py#L120-L126

But then... when it gets to the datasette repository which I have starred it over-writes my full user record with one that's missing most of the details, thanks to this bit:

https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/utils.py#L117-L124

I need to find a way of NOT over-writing a good record with a thinner one.

simonw commented 4 years ago

sqlite-utils supports proper upserts now so this problem should be easy to fix.

simonw commented 4 years ago

I'm abandoning this in favour of a new implementation.