shurcooL / githubv4

Package githubv4 is a client library for accessing GitHub GraphQL API v4 (https://docs.github.com/en/graphql).
MIT License
1.11k stars 89 forks source link

Question: Would a single pagination cursor solve GitHub v4 API's woes? #20

Closed gmlewis closed 7 years ago

gmlewis commented 7 years ago

While playing with the GitHub v4 GraphQL API, we quickly ran up against what appears to be a significant limitation: multiple pagination cursors can not be followed in a single query.

At this point, it is not clear to me if this is a fundamental GraphQL API issue or an issue with the GitHub implementation of the GraphQL API, so I would like to open a discussion here where people with more experience can help clear up any inaccuracies and ideally help propose a solution to GitHub.

Let's paint a hypothetical picture for discussion (but first note that the GitHub v4 GraphQL API limits each entity response to 100 items).

Let's say a large company has ~200 orgs each with an average of ~250 repositories and each of those repos has ~300 contributors (and each contributor has "owner", "write" or "read" privileges).

Let's say I would like to build up a githubql query that answers the question:

"Give me all contributors (and their privileges) of all repositories of all organizations in my account."

Obviously, pagination is needed... but the way it is currently implemented, a pagination cursor is provided for each list of contributors, each list of repositories, and each list of organizations. As a result, it is not possible to complete the query by following a single pagination cursor. Furthermore, it is not clear to me that the query can be completed at all due to the ambiguity of specifying a pagination cursor for one list of contributors for one org/repo combo versus the next org/repo combo.

(I will add an example later, but wanted to keep this as small as possible to highlight the issue with the GitHub v4 GraphQL API.)

Ideally, since a GraphQL query is naturally a depth-first search (since the full depth of the query is specified up-front), there should be a single pagination cursor that can return the paginated results in depth-first order. (As it currently stands, each list is expanded breadth-first with pagination cursors provided for each list that contains over 100 items.)

I will work on putting together an example, but in the meantime, please let me know which portions of this need more explanation or if you would prefer that I move this discussion to the GitHub forums instead.

dmitshur commented 7 years ago

or if you would prefer that I move this discussion to the GitHub forums instead.

If I understand correctly, this is an issue that applies to GitHub GraphQL v4 API itself, and it's not specific to this Go client implementation (i.e., it would apply to a Ruby GraphQL client too).

If so, I definitely think the GraphQL API GitHub forum would be a better fit, since there's a much bigger audience of people (Go users, Ruby users, Java users, etc.) that will see it and they're equally affected too. I'd suggesting moving it there for better visibility.

gmlewis commented 7 years ago

SGTM. Thanks, @shurcooL.

dmitshur commented 6 years ago

@gmlewis Have you already posted in that forum? If so, what's the URL of the post?

gmlewis commented 6 years ago

So sorry, I haven't posted in the forum... I was hoping to spend time reading through the current posts there to see if anyone has come up with this idea and see if it got any feedback.

I've unfortunately let this slip. I'll keep it on my TODO list, though.

Feel free to post if you wish.

dmitshur commented 6 years ago

No problem. Please let me know when you do post it, I am interested in following it for updates.

laike9m commented 6 years ago

Also interested to know about this.

gmlewis commented 6 years ago

I've not done anything more with GraphQL... others are welcome to take over this investigation as my current activities don't include GraphQL.