search

canonical / charm-eng-releases

See all releases and the status of repos for charm engineering at Canonical
https://releases.juju.is
Apache License 2.0
4 stars 8 forks source link

Half of the teams end up with no data #5

Closed simskij closed 1 year ago

simskij commented 1 year ago

Noticed today that half of the teams end up with no data. Currently, the following teams are left blank:

See https://github.com/canonical/charm-eng-releases/blob/8f35a34dc97c965be2f544f6045da5393597dd87/data/repos.json for an example of a broken data file.

jnsgruk commented 1 year ago

Yeh, this is because the number of repos has grown and the tool is hitting a secondary rate limit on GitHub's api.

I did introduce some changes to releasegen last week to try and work around that a bit, but it's not been as successful as I'd hoped 😕

jnsgruk commented 1 year ago

This was the change: https://github.com/jnsgruk/releasegen/commit/3f8dd054af0cafda34f011c069cd42316b7b50c6

I think there is more customisation that can be done on the limiter but I ran out of time last week!

jnsgruk commented 1 year ago

It seems the secondary rate limit is incurred by making concurrent requests - I can stop that behaviour and just accept that the build time will be much longer.

Given that I've already increased the interval between refreshes to avoid the primary rate limit, perhaps that's the best way to go so the page is at least complete?

simskij commented 1 year ago

I think so, at least for the time being. We can discuss other ways of approaching the problem, but it would be great if it at least was complete in the meanwhile.

jnsgruk commented 1 year ago

I'll have a play later this week and see if dropping the concurrency helps resolve the rate limiting

jnsgruk commented 1 year ago

@simskij :crossed_fingers: this is now fixed.

Related commit in releasegen