Fix tap-gitlab's pagination for requests with too many resources

[pnadolny13]

In GitLab by @iroussos on Jun 20, 2019, 05:51

At the moment, when we run tap-gitlab against gitlab-org/gitlab-ce with no date restrictions, we get no results for the following entities:

• issues
• merge_requests
• releases

The reason is discussed in the other pagination headers of Gitlab's API:

Caution: For performance reasons since GitLab 11.8 and behind the api_kaminari_count_with_limit feature flag, if the number of resources is more than 10,000, the X-Total and X-Total-Pages headers as well as the rel="last" Link are not present in the response headers.

tap-gitlab uses the X-Total-Pages and defaults to 1 when it does not find it.

We should:

• update tap-gitlab to use the X-Next-Page header
• increase the per_page param from the default of 20 to its maximum of 100 so that we don't have to send 5000+ requests to get all commits/issues for projects as large as gitlab-org/gitlab-ce

[pnadolny13]

In GitLab by @iroussos on Jun 20, 2019, 07:36

mentioned in merge request !13

[pnadolny13]

In GitLab by @iroussos on Jun 20, 2019, 07:36

closed via merge request !13

[pnadolny13]

In GitLab by @iroussos on Jun 20, 2019, 07:37

mentioned in commit b2a096d35042b3db0d0a69a0af7f51256e8015f4