are these valid fedora infra crawler IPs

[bilias]

Hi, We're seeing thousands of hits on our mirror from 2 IPs. Are these yours?

 Hits      h% Vis.     v% Tx. Amount Data
 ----- ------ ---- ------ ---------- ----
 63566 10.34%    1  0.00%   2.63 MiB 38.145.60.4                             
 24905  4.05%    2  0.00%   1.51 MiB 38.145.60.3                      

38.145.60.3 - - [07/Jun/2024:03:27:10 +0300] "GET /pub/linux/epel/9/Everything/x86_64/repodata/repomd.xml HTTP/1.1" 200 4374 "-" "mirrormanager-crawler/0.1 (+https://github.com/fedora-infra/mirrormanager2/)"
38.145.60.3 - - [07/Jun/2024:08:00:05 +0300] "HEAD /mirrors/linux/centos-stream/9-stream/AppStream/aarch64/os/.discinfo HTTP/1.1" 200 0 "-" "mirrormanager-crawler/0.1 (+https://github.com/fedora-infra/mirrormanager2/)"

38.145.60.4 - - [07/Jun/2024:03:26:59 +0300] "GET /pub/linux/fedora/linux/updates/38/Everything/x86_64/repodata/repomd.xml HTTP/1.1" 404 146 "-" "mirrormanager-crawler/0.1 (+https://github.com/fedora-infra/mirrormanager2/)"
38.145.60.4 - - [07/Jun/2024:03:45:16 +0300] "HEAD /pub/linux/epel/7/aarch64/debug/Packages/c/CUnit-debuginfo-2.1.3-8.el7.aarch64.rpm HTTP/1.1" 200 0 "-" "mirrormanager-crawler/0.1 (+https://github.com/fedora-infra/mirrormanager2/)"

Most of them are HEAD requests. These are too many hits. Is this valid traffic?

[abompard]

Hello! Yes, this is the system that Fedora uses to ensure that the mirrors are up-to-date. An alternative would be to allow an rsync connection from our IP, which will result in a single rsync connection.

https://fedoraproject.org/wiki/Infrastructure/Mirroring#Rsync_Configuration_(sample)

Would that be an option for you?

[bilias]

Hi, We already run rsync on this mirror, and rsync is also registered in Mirrormanager.

I can't find any option in mirrormanager to apply for rsync mirror checking.

[abompard]

@bilias I'll check it, could you please indicate what is the mirror name? (or ID, if you don't want to disclose the name)

[bilias]

site 128 host 488