Closed shr-project closed 3 years ago
For kubectl when reading 100 results per page (https://api.github.com/repos/kubernetes/kubernetes/releases&per_page=100) there are 5 pages as indicated by the returned header:
link: <https://api.github.com/repositories/20580498/releases?per_page=100&page=2>; rel="next", <https://api.github.com/repositories/20580498/releases?per_page=100&page=5>; rel="last"
To preserve some Ratelimit it might be possible to allow partial cache.json updates, that it would read existing cache.json first and then read the pages only as long as there are some new versions which aren't in the cache yet (assuming that github release never really disappears).
Maybe using a library like https://pkg.go.dev/github.com/google/go-github/v37/github#hdr-Pagination which supports Pagination as well as RateLimiting https://pkg.go.dev/github.com/google/go-github/v37/github#hdr-Rate_Limiting.
@shr-project thanks for the heads up on this. I will look into it ASAP.
I think I will handle this here https://github.com/devops-works/binenv/blob/develop/internal/list/github_releases.go#L44 Extracting the HTTP client code from this function and iterate until we have no pages left.
The rate limit is not really an issue, since this is mostly executed using a token in GitHub actions. Given the number of distributions, it is already not feasible to run without a GitHub token, so I'd say this is not an issue (and there is a lot of emphasis about this in the README).
Does this plan sounds good to you @shr-project ?
Sounds good to me.
Thanks for quick fix, verified locally and it works fine.
Awesome :+1: Thanks for reporting !
With kubectl we've recently noticed that 1.19.7 version (which we had in .binenv.lock) disappears when kubectl versions are updated, it's the last one in the cache.json as of: https://github.com/devops-works/binenv/commit/165c1fe4d2eea6a44137d8aed784dec16f585202# but in the next update: https://github.com/devops-works/binenv/commit/c1da30c5abf9f33a23f0df9553350c802833f26d it's pushed out by new 1.22.0-beta.0 release.
This is very unfortunate as it makes impossible to install older locked versions with binenv.
It's caused by github API used in distributions.yaml, e.g. for kubectl: https://api.github.com/repos/kubernetes/kubernetes/releases
which according to https://docs.github.com/en/rest/reference/repos#list-releases returns only 30 results per page.
Easiest work around would be to add per_page=100 to these URLs in ditributions.yaml, but that would only postpone this issue for a while (as some of the tools supported by binenv do release very often).
It would be better to support reading all pages with the results in internal/list/github_releases.go, it might cause the Ratelimit to run out sooner, but supporting all versions (especially for locking) is IMHO worth doing this.
I can try to modify the code to do this, but I'm not very familiar with Go yet. Please confirm that this is worth implementing.