search

helius-labs / validator-firewall

A low level ingress firewall for Solana validators
Other
24 stars 6 forks source link

Dropping packets from allowed IPs #15

Open ferric-sol opened 3 months ago

ferric-sol commented 3 months ago

This may be user error so please tell me to stfu.

My static_overrides.yml is as follows:

(venv) root@host-92-204-168-17:~/validator-firewall# cat /etc/validator-firewall/static_overrides.yml
allow:
  - name: "ashburn"
    ip: 45.43.11.28
deny:

(It wouldn't work without the deny section)

But I'm seeing this in the logs:

(venv) root@host-92-204-168-17:~/validator-firewall# sudo journalctl -u validator-firewall.service -f
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.456256Z","level":"INFO","fields":{"message":"Loaded static overrides: StaticOverrides { allow: [NameAddressPair { name: \"ashburn\", ip: 45.43.11.
28/32 }], deny: [] }","log.target":"validator_firewall","log.module_path":"validator_firewall","log.file":"validator-firewall/src/main.rs","log.line":86},"target":"validator_firewall"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.456284Z","level":"WARN","fields":{"message":"No protected ports provided, defaulting to 8009 and 8010","log.target":"validator_firewall","log.modu
le_path":"validator_firewall","log.file":"validator-firewall/src/main.rs","log.line":92},"target":"validator_firewall"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.629799Z","level":"INFO","fields":{"message":"Filtering UDP ports: [8009, 8010]","log.target":"validator_firewall","log.module_path":"validator_fir
ewall","log.file":"validator-firewall/src/main.rs","log.line":130},"target":"validator_firewall"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.629837Z","level":"WARN","fields":{"message":"No deny list client specified, only using static overrides","log.target":"validator_firewall","log.mo
dule_path":"validator_firewall","log.file":"validator-firewall/src/main.rs","log.line":171},"target":"validator_firewall"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.646037Z","level":"INFO","fields":{"message":"Waiting for Ctrl-C...","log.target":"validator_firewall","log.module_path":"validator_firewall","log.
file":"validator-firewall/src/main.rs","log.line":212},"target":"validator_firewall"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.646114Z","level":"WARN","fields":{"message":"Entering close to leader mode due to missing leader status","log.target":"validator_firewall::leader_
tracker","log.module_path":"validator_firewall::leader_tracker","log.file":"validator-firewall/src/leader_tracker.rs","log.line":277},"target":"validator_firewall::leader_tracker"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.646142Z","level":"INFO","fields":{"message":"All traffic summary: 0 pkts last_interval 0 pkts 0 pkts/s","traffic_type":"All","rate":0,"delta":0,"t
otal":0},"target":"validator_firewall::stats_service"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.646178Z","level":"INFO","fields":{"message":"Blocked traffic summary: 0 pkts last_interval 0 pkts 0 pkts/s","traffic_type":"Blocked","rate":0,"del
ta":0,"total":0},"target":"validator_firewall::stats_service"}
Jul 30 02:11:09 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:09.703070Z","level":"INFO","fields":{"message":"New leader schedule loaded. Epoch 649 max slot 280800000","log.target":"validator_firewall::leader_tr
acker","log.module_path":"validator_firewall::leader_tracker","log.file":"validator-firewall/src/leader_tracker.rs","log.line":86},"target":"validator_firewall::leader_tracker"}
Jul 30 02:11:10 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:10.147163Z","level":"INFO","fields":{"message":"Exiting close to leader mode: Current 87461","log.target":"validator_firewall::leader_tracker","log.m
odule_path":"validator_firewall::leader_tracker","log.file":"validator-firewall/src/leader_tracker.rs","log.line":259},"target":"validator_firewall::leader_tracker"}
Jul 30 02:11:19 host-92-204-168-17.example.com validator-firewall[712972]: {"timestamp":"2024-07-30T02:11:19.648059Z","level":"INFO","fields":{"message":"total_packets: 162.19.222.240 = 38"},"target":"validator_firewall::stats_service"}

...snip...

Jul 30 01:53:11 host-92-204-168-17.example.com validator-firewall[711450]: {"timestamp":"2024-07-30T01:53:11.876920Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 262"},"target":"validator_firewall::stats_service"}

why is it dropping packets from the allow override host? misconfiguration, or am I just missing something?

helius-kurt commented 3 months ago

Hey @ferric-sol can you try with latest from main?

ferric-sol commented 2 months ago

That fixed it, thanks @helius-kurt!

ferric-sol commented 2 months ago

Actually, still happening on one out of two hosts:

Aug 29 03:17:05 host881025 validator-firewall[240432]: {"timestamp":"2024-08-29T03:17:05.583126Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 6"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:05 host881025 validator-firewall[240432]: {"timestamp":"2024-08-29T03:17:05.583276Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 6"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:15 host881025 validator-firewall[240432]: {"timestamp":"2024-08-29T03:17:15.584338Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 17"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:15 host881025 validator-firewall[240432]: {"timestamp":"2024-08-29T03:17:15.584510Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 17"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:25 host881025 validator-firewall[240432]: {"timestamp":"2024-08-29T03:17:25.585806Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 17"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:25 host881025 validator-firewall[240432]: {"timestamp":"2024-08-29T03:17:25.586009Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 17"},"target":"validator_firewall::stats_service"}
^C
root@host881025:~/validator-firewall# systemctl stop validator-firewall
root@host881025:~/validator-firewall# cat /etc/systemd/system/validator-firewall.service
[Unit]
Description=Validator Firewall Service
After=network.target

[Service]
Environment=RUST_LOG=info
ExecStart=/usr/local/sbin/validator-firewall --iface bond0 --static-overrides /etc/validator-firewall/static_overrides.yml
Restart=always

[Install]
WantedBy=multi-user.target
root@host881025:~/validator-firewall# cat /etc/validator-firewall/static_overrides.yml
allow:
  - name: "ashburn"
    ip: 45.43.11.28
deny:

not happening on the other host:

root@ftrx-0009:~/validator-firewall# sudo journalctl -u validator-firewall.service -f | grep 45.43
Aug 29 03:17:06 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:06.156746Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16092"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:06 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:06.157226Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:16 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:16.158195Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16101"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:16 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:16.158756Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:26 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:26.160580Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16111"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:26 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:26.161143Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:36 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:36.162761Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16120"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:36 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:36.163213Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:46 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:46.164079Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16130"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:46 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:46.164561Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:56 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:56.167070Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16139"},"target":"validator_firewall::stats_service"}
Aug 29 03:17:56 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:17:56.167656Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:18:06 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:18:06.169534Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16149"},"target":"validator_firewall::stats_service"}
Aug 29 03:18:06 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:18:06.169971Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:18:16 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:18:16.171628Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16159"},"target":"validator_firewall::stats_service"}
Aug 29 03:18:16 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:18:16.172071Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
Aug 29 03:18:26 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:18:26.173703Z","level":"INFO","fields":{"message":"total_packets: 45.43.11.28 = 16168"},"target":"validator_firewall::stats_service"}
Aug 29 03:18:26 ftrx-0009 validator-firewall[2949575]: {"timestamp":"2024-08-29T03:18:26.174272Z","level":"INFO","fields":{"message":"dropped_packets: 45.43.11.28 = 0"},"target":"validator_firewall::stats_service"}
^C
root@ftrx-0009:~/validator-firewall# cat /etc/validator-firewall/static_overrides.yml
allow:
  - name: "ashburn"
    ip: 45.43.11.28
deny:

thoughts, @helius-kurt ?

StaRkeSolanaValidator commented 2 months ago

Same issue here. Whitelist IPs get denied anyway

StaRkeSolanaValidator commented 1 day ago

Hey @helius-kurt , sorry to ping. Just a head's up that we are still having this issue. Thanks!