dogsheep / github-to-sqlite

Save data from GitHub to a SQLite database
https://github-to-sqlite.dogsheep.net/
Apache License 2.0
402 stars 43 forks source link

Commands for making authenticated API calls #50

Open simonw opened 3 years ago

simonw commented 3 years ago

Similar to twitter-to-sqlite fetch, see https://github.com/dogsheep/twitter-to-sqlite/issues/51

simonw commented 3 years ago

I'm going to start with github-to-sqlite get and github-to-sqlite post - I may add put and suchlike later on.

simonw commented 3 years ago

I'd like a --paginate option that does the same thing as https://github.com/simonw/paginate-json

simonw commented 3 years ago

Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls

simonw commented 3 years ago

Fun demo of the --nl option:

github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl
simonw commented 3 years ago

Bug with endpoints that return dictionaries rather than arrays:

github-to-sqlite get /users/simonw
[
    "login",
    "id",
    "node_id",
    "avatar_url",
    "gravatar_id",
    "url",
    "html_url",
    "followers_url",
    "following_url",
    "gists_url",
    "starred_url",
    "subscriptions_url",
    "organizations_url",
    "repos_url",
    "events_url",
    "received_events_url",
    "type",
    "site_admin",
    "name",
    "company",
    "blog",
    "location",
    "email",
    "hireable",
    "bio",
    "twitter_username",
    "public_repos",
    "public_gists",
    "followers",
    "following",
    "created_at",
    "updated_at"
]
simonw commented 3 years ago

It would be useful if you could pass an --accept option to this.

simonw commented 3 years ago

Demo of --accept:

github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html'