dogsheep / github-to-sqlite

Save data from GitHub to a SQLite database
https://github-to-sqlite.dogsheep.net/
Apache License 2.0
402 stars 43 forks source link

feature: support "events" #64

Open khimaros opened 3 years ago

khimaros commented 3 years ago

the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user

this would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.

khimaros commented 3 years ago

i have a basic working version at https://github.com/khimaros/github-to-sqlite

this can be tested with github-to-sqlite events.db khimaros/events

caveat: the GitHub API doesn't seem to provide a complete history of events.

khimaros commented 3 years ago

it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.

this is significant change, but may be worth considering in the future.

simonw commented 3 years ago

Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2

simonw commented 3 years ago

I'm definitely interested in supporting events in this tool - see #14.

khimaros commented 3 years ago

@simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.

found by poking around in the "Explorer" tab in https://docs.github.com/en/graphql/overview/explorer

note: pagination is still required via first and last but it seems to allow unlimited history.

query MyQuery {
  __typename
  user(login: "<user>") {
    id
    pinnedItems(first: 100) {
      edges {
        node
      }
    }
    pullRequests(first: 100) {
      nodes {
        body
        title
        state
        createdAt
      }
    }
    createdAt
    issues(first: 100) {
      pageInfo {
        endCursor
        startCursor
      }
      nodes {
        title
        url
        createdAt
        body
      }
    }
    issueComments(first: 100) {
      edges {
        node {
          id
          updatedAt
          url
          body
        }
      }
    }
    repositories(first: 100) {
      nodes {
        createdAt
        description
        parent {
          name
        }
        pinnedIssues(first: 100) {
          edges {
            node {
              id
            }
          }
        }
        pinnedDiscussions(first: 100) {
          edges {
            node {
              id
            }
          }
        }
      }
    }
    starredRepositories(first: 100) {
      edges {
        node {
          id
        }
      }
    }
  }
}