release-argus / Argus

Argus is a lightweight monitor to notify of new software releases via Gotify/Slack/other messages and/or WebHooks.
https://release-argus.io
Apache License 2.0
291 stars 11 forks source link

feat: Support for conditional Requests using headers #175

Closed mytlogos closed 1 year ago

mytlogos commented 1 year ago

Is your feature request related to a problem? Please describe. For user who do not want to use a github account, or are tracking many projects, will more or less likely run into the github api rate-limiting, currently at 60 requests/hour or 5000 request/hour/user. Especially for unauthenticated requests, this limit will be hit very quickly.

Describe the solution you'd like Github provides a mechanism to reduce the api limit hit-rate, by using conditional requests. Argus would query github at first time as normal, store the Etag for the service of type github. At the next request for the same service, the header If-None-Match should be use with the previous etag-value. If the resource changed (different etag), the server responds per normal with status code 200, if nothing changed it responds with 304 Not Modified. If nothing was modified, the remaining api limit should not be reduced. In that case argus could just abort with a nothing changed error or similar, or return the response of the provided etag, in case it was cached too (though just "aborting" would be better in my opinion, saves resources)

Example headers from using a command like curl -v https://api.github.com/repos/release-argus/argus/releases

< HTTP/2 200 
< server: GitHub.com
< date: Mon, 14 Nov 2022 15:50:35 GMT
< content-type: application/json; charset=utf-8
< cache-control: public, max-age=60, s-maxage=60
< vary: Accept, Accept-Encoding, Accept, X-Requested-With
< etag: W/"1ce3fe5cd3749c19b72b12dd7c1f2308bd530d49ef555a633eb0ee0b622d07ca"
< x-github-media-type: github.v3; format=json
< link: <https://api.github.com/repositories/145238857/releases?page=2>; rel="next", <https://api.github.com/repositories/145238857/releases?page=6>; rel="last"
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 1
< x-ratelimit-reset: 1668442909
< x-ratelimit-resource: core
< x-ratelimit-used: 59
< accept-ranges: bytes
< x-github-request-id: DBF2:F548:B4CCF1B:B7A996E:637263CB

Example headers from using a command like curl -v -H 'If-None-Match: W/"1ce3fe5cd3749c19b72b12dd7c1f2308bd530d49ef555a633eb0ee0b622d07ca"' https://api.github.com/repos/release-argus/argus/releases (no response body)

< HTTP/2 304 
< server: GitHub.com
< date: Mon, 14 Nov 2022 16:38:41 GMT
< content-type: application/json; charset=utf-8
< cache-control: public, max-age=60, s-maxage=60
< vary: Accept, Accept-Encoding, Accept, X-Requested-With
< etag: W/"1ce3fe5cd3749c19b72b12dd7c1f2308bd530d49ef555a633eb0ee0b622d07ca"
< x-github-media-type: github.v3; format=json
< link: <https://api.github.com/repositories/145238857/releases?page=2>; rel="next", <https://api.github.com/repositories/145238857/releases?page=6>; rel="last"
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 21
< x-ratelimit-reset: 1668446515
< x-ratelimit-resource: core
< x-ratelimit-used: 39
< x-github-request-id: E606:75C1:B8F5770:BBCEA34:63726F11
< 

Describe alternatives you've considered The only alternative is to use a github account and hope.

Additional context This would be a huge saver for all users, as argus would most likely never hit the api rate limit again, at least in github (just being optimistic).

JosephKav commented 1 year ago

Oh cool, didn't know about this 'conditional requests' feature. I'll get straight on this! Thanks for letting me know about it, it will definitely help with the rate-limiting!

I'll get it storing the ETag each time it queries and use it if it has one. I'll also store the filtered release tags until the tag at the top (most-recent) matches the require filters as

JosephKav commented 1 year ago

Right, I've got ETag's working in my PR. This change definitely reduces the number of calls that count towards the API limit, but from my testing of 18 services with a 1m interval, between 4 and 6 of them had their ETag change every time (usually the same services too). Not 100% sure on the reasoning, perhaps people reacting to the releases? I'll merge my PR shortly, just letting you know incase you see the x-ratelimit-remaining as less than you'd expect

(my PR won't print the 'changed from X to Y' part, I just added that to have a look at how different the ETags were)

Nov 15 15:26:25: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "" to "\"1cce2be4ac290afc04dd3e812b736124b0f9c85248457a91f3b6e6682456619d\"")
Nov 15 15:26:25: VERBOSE: goauthentik/authentik, Potentially found new releases (ETag changed from "" to "\"c9fce6e745b813ca728d6a9165b4856becea9d1e467e2f42f3cb46216a03073e\"")
Nov 15 15:26:26: VERBOSE: ansible/awx-operator, Potentially found new releases (ETag changed from "" to "\"074370e5c432c5c183b0d09b8a9fbf143c05529f0f524b230b653bebbe47d569\"")
Nov 15 15:26:29: VERBOSE: grafana/grafana, Potentially found new releases (ETag changed from "" to "\"201dbb01c138b8c5c94401afebdce5af0fcdcb6dadc082d8baa4a14b2d84e8f0\"")
Nov 15 15:26:29: VERBOSE: go-gitea/gitea, Potentially found new releases (ETag changed from "" to "\"22bffd1a01a8c3bfab2d0e073aa443216426ffb2f87f012383a448a3a069ac62\"")
Nov 15 15:26:29: VERBOSE: gotify/server, Potentially found new releases (ETag changed from "" to "\"cdea08072b02a50412cf705350c5f1ff5aaffd393aa2f73f20387243d2094f2f\"")
Nov 15 15:26:30: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "" to "\"f81730cbc981d75876d3cf7fe56dd1109f04a8fb4353134a8f60dde443f31fac\"")
Nov 15 15:26:30: VERBOSE: home-assistant/core, Potentially found new releases (ETag changed from "" to "\"e2fd6b450f8a936ea7f947ba5ec3c428235e360ab9ce05a2288c0449d02fdc39\"")
Nov 15 15:26:31: VERBOSE: mailcow/mailcow-dockerized, Potentially found new releases (ETag changed from "" to "\"e726925ffa36c2b40c8e37a8bff00025086d54320adbf4673398f32c2c4eca47\"")
Nov 15 15:26:32: VERBOSE: nextcloud/server, Potentially found new releases (ETag changed from "" to "\"06cc5d4eea589bf8a1a8eeae3d70c7bc22f01d904cb9933111bfd8426ac9de85\"")
Nov 15 15:26:33: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "" to "\"b04cb854480aced98c9f289d690b570a0323aff7650a38760d13eb72bdf9ed5b\"")
Nov 15 15:26:33: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "" to "\"710f7debcba9e6ba446d2478357a8640209527eb10a5ceb8f9fc862ba23e485c\"")
Nov 15 15:26:34: VERBOSE: containrrr/shoutrrr, Potentially found new releases (ETag changed from "" to "\"869d3391b2f3c1f4d4bf2a761bc9da51593e9d24787033f98861d3f03d106a5e\"")
Nov 15 15:26:34: VERBOSE: thanos-io/thanos, Potentially found new releases (ETag changed from "" to "\"578f1e5b1fa29fa48bd30fd68ce8175764a402d7231fac65226309f643d0eeef\"")
Nov 15 15:26:35: VERBOSE: louislam/uptime-kuma, Potentially found new releases (ETag changed from "" to "\"274ea11913d95cc8f41325c073151f2ef533c9bbb54c96d5cd172b6c550a53fa\"")
Nov 15 15:26:35: VERBOSE: dani-garcia/vaultwarden, Potentially found new releases (ETag changed from "" to "\"639656e52938b2e307f709e879ecb3f826fe055f89178607b8240edc82ed72b5\"")
Nov 15 15:26:36: VERBOSE: adnanh/webhook, Potentially found new releases (ETag changed from "" to "\"2c6ce9a8358e10a6dd958db5d34de58222b8ba9c053e84d835b8211ff1bd74cb\"")
Nov 15 15:26:37: VERBOSE: wowchemy/wowchemy-hugo-themes, Potentially found new releases (ETag changed from "" to "\"217673ec8a05c6bbb659e735eaa508a070e214e34a62e660de3c58fc47d3b075\"")

Nov 15 15:27:26: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"1cce2be4ac290afc04dd3e812b736124b0f9c85248457a91f3b6e6682456619d\"" to "\"a013264499ed2f1f6e06b0f6dbd79218bd35a3e133dfcef8882f3267d1810f1e\"")
Nov 15 15:27:31: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"f81730cbc981d75876d3cf7fe56dd1109f04a8fb4353134a8f60dde443f31fac\"" to "\"cc7d097e39a37eb2e8dff1c6f98a93c0f6692262a462dd6d77ceeb3223c922b8\"")
Nov 15 15:27:34: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"710f7debcba9e6ba446d2478357a8640209527eb10a5ceb8f9fc862ba23e485c\"" to "\"c0c934cf2bdc2be5348190dc399f9e1d8eda1e81b9112ab2be609f41be9967e5\"")
Nov 15 15:27:35: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"b04cb854480aced98c9f289d690b570a0323aff7650a38760d13eb72bdf9ed5b\"" to "\"44b6392a0a3eec28e02bd9333ba22b93b56f84deb730a7f832cfea9881913295\"")
Nov 15 15:27:36: VERBOSE: louislam/uptime-kuma, Potentially found new releases (ETag changed from "\"274ea11913d95cc8f41325c073151f2ef533c9bbb54c96d5cd172b6c550a53fa\"" to "\"a350811d0893ad75fa29574d76267dba5619e8874190644f156d1ba8c5997f40\"")

Nov 15 15:28:27: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"a013264499ed2f1f6e06b0f6dbd79218bd35a3e133dfcef8882f3267d1810f1e\"" to "\"34aa364c0dbd4c90b06d5d1301ea8cf64d84c636664a1a0ad9d50adfcc40637e\"")
Nov 15 15:28:33: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"cc7d097e39a37eb2e8dff1c6f98a93c0f6692262a462dd6d77ceeb3223c922b8\"" to "\"722c11115fecf62721c5bd07ce32ffe63bbc27f8c701ef94501350aa4ba6e967\"")
Nov 15 15:28:36: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"c0c934cf2bdc2be5348190dc399f9e1d8eda1e81b9112ab2be609f41be9967e5\"" to "\"741c95b222af471b1aed38db757a9074468dc0c1675fd31e7595ac2b732123f4\"")
Nov 15 15:28:37: VERBOSE: louislam/uptime-kuma, Potentially found new releases (ETag changed from "\"a350811d0893ad75fa29574d76267dba5619e8874190644f156d1ba8c5997f40\"" to "\"371e53fc6142dedee8b2f3062420d5c1f45e4dc1029082bfa8cde3dbfb562d9b\"")
Nov 15 15:28:37: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"44b6392a0a3eec28e02bd9333ba22b93b56f84deb730a7f832cfea9881913295\"" to "\"388c1e53771efb0dfc1a593f6d2372428c162430e9958af43530937901b6c645\"")

Nov 15 15:29:28: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"34aa364c0dbd4c90b06d5d1301ea8cf64d84c636664a1a0ad9d50adfcc40637e\"" to "\"d69c1bf949f0ffa2b47c5485daf40a32451a1dc6326dbc2040f0229352a78462\"")
Nov 15 15:29:28: VERBOSE: ansible/awx-operator, Potentially found new releases (ETag changed from "\"074370e5c432c5c183b0d09b8a9fbf143c05529f0f524b230b653bebbe47d569\"" to "\"7700b5e4a8ea13ac9fa70c393d35195513d02c1034352c816adc996924b6f103\"")
Nov 15 15:29:34: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"722c11115fecf62721c5bd07ce32ffe63bbc27f8c701ef94501350aa4ba6e967\"" to "\"9777c63d8393014a44e850bf5f6efbe33ec1f07b797b557cf7a72f02b0edfc7c\"")
Nov 15 15:29:36: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"741c95b222af471b1aed38db757a9074468dc0c1675fd31e7595ac2b732123f4\"" to "\"4757c5c09fc7703b746993cd9f82437c732dcea69f4cabd0de1886d99b58f793\"")
Nov 15 15:29:40: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"388c1e53771efb0dfc1a593f6d2372428c162430e9958af43530937901b6c645\"" to "\"b3d2b729a7df8b92a55ea7411449e755af775a7eab28d67af5108802bd8907a4\"")

Nov 15 15:30:29: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"d69c1bf949f0ffa2b47c5485daf40a32451a1dc6326dbc2040f0229352a78462\"" to "\"315158f33db861b21faf37d62ce6dc9ac63c9e9532f84566ed32ea4d369e9de5\"")
Nov 15 15:30:35: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"9777c63d8393014a44e850bf5f6efbe33ec1f07b797b557cf7a72f02b0edfc7c\"" to "\"8aaf10519d6cd33b035281ae56e0a3e714314808c0a746d07d1cb4c98dd68c9f\"")
Nov 15 15:30:37: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"4757c5c09fc7703b746993cd9f82437c732dcea69f4cabd0de1886d99b58f793\"" to "\"1f4b97a2dbdec2522cbb4c2d820b8b62d2a436cb64d7ee7b10d380fa7e9284f4\"")
Nov 15 15:30:41: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"b3d2b729a7df8b92a55ea7411449e755af775a7eab28d67af5108802bd8907a4\"" to "\"98fc1beacc168e2c0f231c8024528d562ec32e95709a0962cfb5b0804927387f\"")

Nov 15 15:31:32: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"315158f33db861b21faf37d62ce6dc9ac63c9e9532f84566ed32ea4d369e9de5\"" to "\"2ff8e37fae15c2bb24f4289f7caf8a1a0a3c15ece8a10eb8aa60edef49758662\"")
Nov 15 15:31:36: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"8aaf10519d6cd33b035281ae56e0a3e714314808c0a746d07d1cb4c98dd68c9f\"" to "\"579e11418b039990077162f9505d9a9aef99c19e6c8cf3b3f818a734b989de4b\"")
Nov 15 15:31:39: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"1f4b97a2dbdec2522cbb4c2d820b8b62d2a436cb64d7ee7b10d380fa7e9284f4\"" to "\"be6a9c821bf14331bcbd6c2af5d2c00451030fedf9f50d74000cad2e124d5eee\"")
Nov 15 15:31:43: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"98fc1beacc168e2c0f231c8024528d562ec32e95709a0962cfb5b0804927387f\"" to "\"c9e9140cb5ab8ab7d366606388f87d80c58eac964812ce7a8e5788e77e0a431f\"")

Nov 15 15:32:31: VERBOSE: ansible/awx-operator, Potentially found new releases (ETag changed from "\"7700b5e4a8ea13ac9fa70c393d35195513d02c1034352c816adc996924b6f103\"" to "\"11da6eeb9c8da516545b5df68197d894c56adbdeef0a1283ff9dc2d6be0b9895\"")
Nov 15 15:32:34: VERBOSE: prometheus/alertmanager, Potentially found new releases (ETag changed from "\"2ff8e37fae15c2bb24f4289f7caf8a1a0a3c15ece8a10eb8aa60edef49758662\"" to "\"a4bbf57514ea4cf1239bbdc8a3c88e5d1ec3d33ebe69bc0f1a832191e0c414c0\"")
Nov 15 15:32:38: VERBOSE: gohugoio/hugo, Potentially found new releases (ETag changed from "\"579e11418b039990077162f9505d9a9aef99c19e6c8cf3b3f818a734b989de4b\"" to "\"7a59871abb16bfbc9a4b04358a90e13a80671b097920967ceb70a795800f54be\"")
Nov 15 15:32:39: VERBOSE: thanos-io/thanos, Potentially found new releases (ETag changed from "\"578f1e5b1fa29fa48bd30fd68ce8175764a402d7231fac65226309f643d0eeef\"" to "\"26d0223733cd2d7906c5941d7c3eca282542637458fee42ce967d9d8014529b7\"")
Nov 15 15:32:40: VERBOSE: grafana/loki, Potentially found new releases (ETag changed from "\"be6a9c821bf14331bcbd6c2af5d2c00451030fedf9f50d74000cad2e124d5eee\"" to "\"09d27d4b8d980cbc1ace1a73e47d9866c4f86f85dc773953f3457e3cbb833e72\"")
Nov 15 15:32:44: VERBOSE: prometheus/prometheus, Potentially found new releases (ETag changed from "\"c9e9140cb5ab8ab7d366606388f87d80c58eac964812ce7a8e5788e77e0a431f\"" to "\"cf80965e1d38589a6b09f80303db4bb48b20b5d9b60bd6109df31f1a1c3c20e0\"")