Install, configure, and tune fail2ban

[acozine]

We have installed fail2ban on our load balancers from the branch for #4834.

Right now most problematic traffic is caught by the rate limits in the catalog, but fail2ban seems like a good tool to have at our disposal, so we have agreed to keep it.

Current known work:

• [ ] refine #4834 (change the tasks and/or tests) so CI passes
• [ ] decide if we want the fail2ban tag and if so, where it should go (possible alternative - a fail2ban playbook?)
• [ ] Investigate why bans don’t seem to affect the browser that the person used to get banned
• [ ] Look at terminology - can we get rid of the word “jail” in our config?
• [ ] Set up monitoring for fail2ban/ufw (send /var/log/fail2ban.log to Datadog, add a check for fail2ban-client, etc.)

[acozine]

When we tested our fail2ban setup by getting ourselves banned, we noticed that new connections were successfully banned but existing ones could continue to make requests and get responses.

Here's the signature in the fail2ban logs when an IP gets banned but is still making requests:

2024-04-09 00:06:24,376 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 35.178.116.124 - 2024-04-09 00:06:24
2024-04-09 18:40:23,623 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:23
2024-04-09 18:40:23,704 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:23
2024-04-09 18:40:23,793 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:23
2024-04-09 18:40:23,881 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:23
2024-04-09 18:40:23,968 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:23
2024-04-09 18:40:24,061 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:24
2024-04-09 18:40:24,276 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:24
2024-04-09 18:40:24,376 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:24
2024-04-09 18:40:25,385 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:25
2024-04-09 18:40:25,428 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:25
2024-04-09 18:40:25,477 fail2ban.actions        [3443249]: NOTICE  [nginx-limit-req] Ban 89.187.164.150
2024-04-09 18:40:27,459 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:27
2024-04-09 18:40:27,549 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:27
2024-04-09 18:40:27,636 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:27
2024-04-09 18:40:27,723 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:27
2024-04-09 18:40:27,824 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:27
2024-04-09 18:40:27,925 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:27
2024-04-09 18:40:28,037 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,340 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,445 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,561 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,650 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,743 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,832 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:28,882 fail2ban.actions        [3443249]: NOTICE  [nginx-limit-req] 89.187.164.150 already banned
2024-04-09 18:40:28,934 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:28
2024-04-09 18:40:29,048 fail2ban.filter         [3443249]: INFO    [nginx-limit-req] Found 89.187.164.150 - 2024-04-09 18:40:29

[acozine]

Related to https://github.com/PrincetonUniversityLibrary/security/issues/75