Closed mytlogos closed 1 year ago
Oh cool, didn't know about this 'conditional requests' feature. I'll get straight on this! Thanks for letting me know about it, it will definitely help with the rate-limiting!
I'll get it storing the ETag each time it queries and use it if it has one. I'll also store the filtered release tags until the tag at the top (most-recent) matches the require
filters as
Right, I've got ETag's working in my PR. This change definitely reduces the number of calls that count towards the API limit, but from my testing of 18 services with a 1m interval, between 4 and 6 of them had their ETag change every time (usually the same services too). Not 100% sure on the reasoning, perhaps people reacting to the releases? I'll merge my PR shortly, just letting you know incase you see the x-ratelimit-remaining
as less than you'd expect
(my PR won't print the 'changed from X to Y' part, I just added that to have a look at how different the ETags were)
Nov 15 15:26:25: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "" to "\"1cce2be4ac290afc04dd3e812b736124b0f9c85248457a91f3b6e6682456619d\"")
Nov 15 15:26:25: VERBOSE: goauthentik/authentik, Potentially found new releases (ETag changed from "" to "\"c9fce6e745b813ca728d6a9165b4856becea9d1e467e2f42f3cb46216a03073e\"")
Nov 15 15:26:26: VERBOSE: ansible/awx-operator, Potentially found new releases (ETag changed from "" to "\"074370e5c432c5c183b0d09b8a9fbf143c05529f0f524b230b653bebbe47d569\"")
Nov 15 15:26:29: VERBOSE: grafana/grafana, Potentially found new releases (ETag changed from "" to "\"201dbb01c138b8c5c94401afebdce5af0fcdcb6dadc082d8baa4a14b2d84e8f0\"")
Nov 15 15:26:29: VERBOSE: go-gitea/gitea, Potentially found new releases (ETag changed from "" to "\"22bffd1a01a8c3bfab2d0e073aa443216426ffb2f87f012383a448a3a069ac62\"")
Nov 15 15:26:29: VERBOSE: gotify/server, Potentially found new releases (ETag changed from "" to "\"cdea08072b02a50412cf705350c5f1ff5aaffd393aa2f73f20387243d2094f2f\"")
Nov 15 15:26:30: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "" to "\"f81730cbc981d75876d3cf7fe56dd1109f04a8fb4353134a8f60dde443f31fac\"")
Nov 15 15:26:30: VERBOSE: home-assistant/core, Potentially found new releases (ETag changed from "" to "\"e2fd6b450f8a936ea7f947ba5ec3c428235e360ab9ce05a2288c0449d02fdc39\"")
Nov 15 15:26:31: VERBOSE: mailcow/mailcow-dockerized, Potentially found new releases (ETag changed from "" to "\"e726925ffa36c2b40c8e37a8bff00025086d54320adbf4673398f32c2c4eca47\"")
Nov 15 15:26:32: VERBOSE: nextcloud/server, Potentially found new releases (ETag changed from "" to "\"06cc5d4eea589bf8a1a8eeae3d70c7bc22f01d904cb9933111bfd8426ac9de85\"")
Nov 15 15:26:33: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "" to "\"b04cb854480aced98c9f289d690b570a0323aff7650a38760d13eb72bdf9ed5b\"")
Nov 15 15:26:33: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "" to "\"710f7debcba9e6ba446d2478357a8640209527eb10a5ceb8f9fc862ba23e485c\"")
Nov 15 15:26:34: VERBOSE: containrrr/shoutrrr, Potentially found new releases (ETag changed from "" to "\"869d3391b2f3c1f4d4bf2a761bc9da51593e9d24787033f98861d3f03d106a5e\"")
Nov 15 15:26:34: VERBOSE: thanos-io/thanos, Potentially found new releases (ETag changed from "" to "\"578f1e5b1fa29fa48bd30fd68ce8175764a402d7231fac65226309f643d0eeef\"")
Nov 15 15:26:35: VERBOSE: louislam/uptime-kuma, Potentially found new releases (ETag changed from "" to "\"274ea11913d95cc8f41325c073151f2ef533c9bbb54c96d5cd172b6c550a53fa\"")
Nov 15 15:26:35: VERBOSE: dani-garcia/vaultwarden, Potentially found new releases (ETag changed from "" to "\"639656e52938b2e307f709e879ecb3f826fe055f89178607b8240edc82ed72b5\"")
Nov 15 15:26:36: VERBOSE: adnanh/webhook, Potentially found new releases (ETag changed from "" to "\"2c6ce9a8358e10a6dd958db5d34de58222b8ba9c053e84d835b8211ff1bd74cb\"")
Nov 15 15:26:37: VERBOSE: wowchemy/wowchemy-hugo-themes, Potentially found new releases (ETag changed from "" to "\"217673ec8a05c6bbb659e735eaa508a070e214e34a62e660de3c58fc47d3b075\"")
Nov 15 15:27:26: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"1cce2be4ac290afc04dd3e812b736124b0f9c85248457a91f3b6e6682456619d\"" to "\"a013264499ed2f1f6e06b0f6dbd79218bd35a3e133dfcef8882f3267d1810f1e\"")
Nov 15 15:27:31: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"f81730cbc981d75876d3cf7fe56dd1109f04a8fb4353134a8f60dde443f31fac\"" to "\"cc7d097e39a37eb2e8dff1c6f98a93c0f6692262a462dd6d77ceeb3223c922b8\"")
Nov 15 15:27:34: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"710f7debcba9e6ba446d2478357a8640209527eb10a5ceb8f9fc862ba23e485c\"" to "\"c0c934cf2bdc2be5348190dc399f9e1d8eda1e81b9112ab2be609f41be9967e5\"")
Nov 15 15:27:35: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"b04cb854480aced98c9f289d690b570a0323aff7650a38760d13eb72bdf9ed5b\"" to "\"44b6392a0a3eec28e02bd9333ba22b93b56f84deb730a7f832cfea9881913295\"")
Nov 15 15:27:36: VERBOSE: louislam/uptime-kuma, Potentially found new releases (ETag changed from "\"274ea11913d95cc8f41325c073151f2ef533c9bbb54c96d5cd172b6c550a53fa\"" to "\"a350811d0893ad75fa29574d76267dba5619e8874190644f156d1ba8c5997f40\"")
Nov 15 15:28:27: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"a013264499ed2f1f6e06b0f6dbd79218bd35a3e133dfcef8882f3267d1810f1e\"" to "\"34aa364c0dbd4c90b06d5d1301ea8cf64d84c636664a1a0ad9d50adfcc40637e\"")
Nov 15 15:28:33: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"cc7d097e39a37eb2e8dff1c6f98a93c0f6692262a462dd6d77ceeb3223c922b8\"" to "\"722c11115fecf62721c5bd07ce32ffe63bbc27f8c701ef94501350aa4ba6e967\"")
Nov 15 15:28:36: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"c0c934cf2bdc2be5348190dc399f9e1d8eda1e81b9112ab2be609f41be9967e5\"" to "\"741c95b222af471b1aed38db757a9074468dc0c1675fd31e7595ac2b732123f4\"")
Nov 15 15:28:37: VERBOSE: louislam/uptime-kuma, Potentially found new releases (ETag changed from "\"a350811d0893ad75fa29574d76267dba5619e8874190644f156d1ba8c5997f40\"" to "\"371e53fc6142dedee8b2f3062420d5c1f45e4dc1029082bfa8cde3dbfb562d9b\"")
Nov 15 15:28:37: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"44b6392a0a3eec28e02bd9333ba22b93b56f84deb730a7f832cfea9881913295\"" to "\"388c1e53771efb0dfc1a593f6d2372428c162430e9958af43530937901b6c645\"")
Nov 15 15:29:28: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"34aa364c0dbd4c90b06d5d1301ea8cf64d84c636664a1a0ad9d50adfcc40637e\"" to "\"d69c1bf949f0ffa2b47c5485daf40a32451a1dc6326dbc2040f0229352a78462\"")
Nov 15 15:29:28: VERBOSE: ansible/awx-operator, Potentially found new releases (ETag changed from "\"074370e5c432c5c183b0d09b8a9fbf143c05529f0f524b230b653bebbe47d569\"" to "\"7700b5e4a8ea13ac9fa70c393d35195513d02c1034352c816adc996924b6f103\"")
Nov 15 15:29:34: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"722c11115fecf62721c5bd07ce32ffe63bbc27f8c701ef94501350aa4ba6e967\"" to "\"9777c63d8393014a44e850bf5f6efbe33ec1f07b797b557cf7a72f02b0edfc7c\"")
Nov 15 15:29:36: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"741c95b222af471b1aed38db757a9074468dc0c1675fd31e7595ac2b732123f4\"" to "\"4757c5c09fc7703b746993cd9f82437c732dcea69f4cabd0de1886d99b58f793\"")
Nov 15 15:29:40: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"388c1e53771efb0dfc1a593f6d2372428c162430e9958af43530937901b6c645\"" to "\"b3d2b729a7df8b92a55ea7411449e755af775a7eab28d67af5108802bd8907a4\"")
Nov 15 15:30:29: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"d69c1bf949f0ffa2b47c5485daf40a32451a1dc6326dbc2040f0229352a78462\"" to "\"315158f33db861b21faf37d62ce6dc9ac63c9e9532f84566ed32ea4d369e9de5\"")
Nov 15 15:30:35: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"9777c63d8393014a44e850bf5f6efbe33ec1f07b797b557cf7a72f02b0edfc7c\"" to "\"8aaf10519d6cd33b035281ae56e0a3e714314808c0a746d07d1cb4c98dd68c9f\"")
Nov 15 15:30:37: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"4757c5c09fc7703b746993cd9f82437c732dcea69f4cabd0de1886d99b58f793\"" to "\"1f4b97a2dbdec2522cbb4c2d820b8b62d2a436cb64d7ee7b10d380fa7e9284f4\"")
Nov 15 15:30:41: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"b3d2b729a7df8b92a55ea7411449e755af775a7eab28d67af5108802bd8907a4\"" to "\"98fc1beacc168e2c0f231c8024528d562ec32e95709a0962cfb5b0804927387f\"")
Nov 15 15:31:32: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"315158f33db861b21faf37d62ce6dc9ac63c9e9532f84566ed32ea4d369e9de5\"" to "\"2ff8e37fae15c2bb24f4289f7caf8a1a0a3c15ece8a10eb8aa60edef49758662\"")
Nov 15 15:31:36: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"8aaf10519d6cd33b035281ae56e0a3e714314808c0a746d07d1cb4c98dd68c9f\"" to "\"579e11418b039990077162f9505d9a9aef99c19e6c8cf3b3f818a734b989de4b\"")
Nov 15 15:31:39: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"1f4b97a2dbdec2522cbb4c2d820b8b62d2a436cb64d7ee7b10d380fa7e9284f4\"" to "\"be6a9c821bf14331bcbd6c2af5d2c00451030fedf9f50d74000cad2e124d5eee\"")
Nov 15 15:31:43: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"98fc1beacc168e2c0f231c8024528d562ec32e95709a0962cfb5b0804927387f\"" to "\"c9e9140cb5ab8ab7d366606388f87d80c58eac964812ce7a8e5788e77e0a431f\"")
Nov 15 15:32:31: VERBOSE: ansible/awx-operator, Potentially found new releases (ETag changed from "\"7700b5e4a8ea13ac9fa70c393d35195513d02c1034352c816adc996924b6f103\"" to "\"11da6eeb9c8da516545b5df68197d894c56adbdeef0a1283ff9dc2d6be0b9895\"")
Nov 15 15:32:34: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"2ff8e37fae15c2bb24f4289f7caf8a1a0a3c15ece8a10eb8aa60edef49758662\"" to "\"a4bbf57514ea4cf1239bbdc8a3c88e5d1ec3d33ebe69bc0f1a832191e0c414c0\"")
Nov 15 15:32:38: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"579e11418b039990077162f9505d9a9aef99c19e6c8cf3b3f818a734b989de4b\"" to "\"7a59871abb16bfbc9a4b04358a90e13a80671b097920967ceb70a795800f54be\"")
Nov 15 15:32:39: VERBOSE: thanos-io/thanos, Potentially found new releases (ETag changed from "\"578f1e5b1fa29fa48bd30fd68ce8175764a402d7231fac65226309f643d0eeef\"" to "\"26d0223733cd2d7906c5941d7c3eca282542637458fee42ce967d9d8014529b7\"")
Nov 15 15:32:40: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"be6a9c821bf14331bcbd6c2af5d2c00451030fedf9f50d74000cad2e124d5eee\"" to "\"09d27d4b8d980cbc1ace1a73e47d9866c4f86f85dc773953f3457e3cbb833e72\"")
Nov 15 15:32:44: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"c9e9140cb5ab8ab7d366606388f87d80c58eac964812ce7a8e5788e77e0a431f\"" to "\"cf80965e1d38589a6b09f80303db4bb48b20b5d9b60bd6109df31f1a1c3c20e0\"")
Is your feature request related to a problem? Please describe. For user who do not want to use a github account, or are tracking many projects, will more or less likely run into the github api rate-limiting, currently at 60 requests/hour or 5000 request/hour/user. Especially for unauthenticated requests, this limit will be hit very quickly.
Describe the solution you'd like Github provides a mechanism to reduce the api limit hit-rate, by using conditional requests. Argus would query github at first time as normal, store the
Etag
for the service of type github. At the next request for the same service, the header If-None-Match should be use with the previous etag-value. If the resource changed (different etag), the server responds per normal with status code 200, if nothing changed it responds with304 Not Modified
. If nothing was modified, the remaining api limit should not be reduced. In that case argus could just abort with anothing changed
error or similar, or return the response of the provided etag, in case it was cached too (though just "aborting" would be better in my opinion, saves resources)Example headers from using a command like
curl -v https://api.github.com/repos/release-argus/argus/releases
Example headers from using a command like
curl -v -H 'If-None-Match: W/"1ce3fe5cd3749c19b72b12dd7c1f2308bd530d49ef555a633eb0ee0b622d07ca"' https://api.github.com/repos/release-argus/argus/releases
(no response body)Describe alternatives you've considered The only alternative is to use a github account and hope.
Additional context This would be a huge saver for all users, as argus would most likely never hit the api rate limit again, at least in github (just being optimistic).