fedora-infra / anitya

A cross-distribution upstream release monitoring project
https://release-monitoring.org
GNU General Public License v2.0
250 stars 105 forks source link

Allow rate-limiting for the custom backend #853

Open zaitseff opened 5 years ago

zaitseff commented 5 years ago

On https://release-monitoring.org/ when editing a project's details, it would be good to have a rate-limiting option for the custom backend. In particular, my project "trader" (https://release-monitoring.org/project/15397/) is currently scanned irregularly roughly every two to twenty minutes -- sometimes the gap is an hour or more, but sometimes it's just every few minutes. This seems rather excessive for a project that is released once a year or so! Providing an option like "scan once a day, once a week, once a month" (and shorter variants, too, for frequently updated projects) would be good.

For context, here is the Apache access log for 15th November:

tx-102-181-132-209.redhat.com - - [15/Nov/2019:00:09:47 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:00:57:09 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:01:21:12 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:01:59:36 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:02:32:11 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:03:44:21 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:04:55:04 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:05:16:18 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:06:06:04 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:06:26:56 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:07:17:12 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:07:32:21 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:08:27:37 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:08:51:24 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:09:38:02 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:10:48:30 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:11:11:18 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:11:59:03 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:12:39:42 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:13:10:22 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:14:22:51 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:14:26:13 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:15:33:26 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:16:41:38 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:16:48:45 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:17:49:51 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:18:50:32 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:18:55:23 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:20:10:55 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:21:26:07 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:21:31:10 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:22:42:19 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
tx-102-181-132-209.redhat.com - - [15/Nov/2019:23:58:54 +1100] "GET /pub/trader/unix/ HTTP/1.1" 200 32817 "-" "Anitya 0.17.2 at release-monitoring.org" TLSv1.3 TLS_AES_256_GCM_SHA384
Zlopez commented 5 years ago

This is strange, the project should be checked once per hour, and I already implemented https://github.com/release-monitoring/anitya/issues/730, which should lower the amount of data to transfer. But it would be nice to have ability to choose how often the project is checked for new versions.