Guide shared by jondurbin

To be reviewed and integrated

https://discord.com/channels/799672011265015819/1191833510021955695/1237409837101482015

Try this:

Create 3+ ec2 instances in AWS in the region closest to your GPU instances, using a different AZ for each with elastic IPs assigned so they are fixed IPs for firewall rules later
Create a network load balancer in the same region, with target groups pointing to the port(s) you'll use for your miners (or different ports, but it can get confusing quickly), select all of the ec2 instances as targets with TCP mode.

On each of the ec2 instances, set up nginx with a server block that does a proxy_pass to your actual instance, e.g.:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 1024;
}
http {
ignore_invalid_headers off;
client_max_body_size 0;
log_format proxylog '$remote_addr - $remote_user upstream=$upstream_addr [$time_local] ($request_time) '
            '"$request" $status $body_bytes_sent '
            '"$http_referer" "$http_user_agent"';
proxy_read_timeout 95s;
upstream miner {
    server a.b.c.d:port;
    server d.e.f.g:port;
}
server {
    access_log /var/log/nginx/access.log proxylog;
    listen 2000-2007;
    location / {
        proxy_pass http://a.b.c.d:$server_port;
    }
}
}

on the miners, also be sure to enable UFW and disable all traffic except inbound from your home IP (or however you login) and the load balancers, e.g. ufw allow proto tcp from a.b.c.d port 2000

run subtensor somewhere else, like a cheap CPU only instance (preferably 3 or more to prevent downtime), then setup nginx on the miner with a proxy to that subtensor instance, something like:

upstream subtensor {
    server a.b.c.d:9944;
    server e.f.g.h:9944 backup;
    server i.j.k.l:9944 backup;
}
server {
    listen 9944;
    location / {
        proxy_pass                http://subtensor;
        proxy_next_upstream       error timeout invalid_header http_500 http_502;
        proxy_next_upstream_tries 5;
        proxy_connect_timeout     1500ms;
        proxy_http_version        1.1;
        proxy_set_header          Upgrade $http_upgrade;
        proxy_set_header          Connection "Upgrade";
        proxy_set_header          Host $host;
        proxy_read_timeout        7d;
    }
}

create an AWS global accelerator that points to the network load balancer, ensuring you have listeners for each of the corresponding listeners on the load balancer
on the miners, be sure to set --axon.external_ip (global accelerator IP) and --axon.external_port to the global accelerator IP (if it doesn't match the port on the gpu instance) (and be sure to modify the code base to ensure axon init passes the entire config, something this repo does not do, e.g. self.axon = ...(..., config=self.config)
run more than one instance of each miner, using the same --axon.external_ip and --axon.external_port so it doesn't try to register a new axon ip/port, do the load balancing in the ec2 instance nginx as in the example above

Update your open file limit and experiment with client_body_timeout, client_header_timeout, and send_timeout to reduce the likelihood of a slowloris type of attack. Also optionally update the location directive to only allow the specific synapses for that miner. If you really want to get fancy you could also continuously query the metagraph for list of validator IPs and update the location directive with allow and deny directives, i.e. deny everything not coming from validator IP.

inference-labs-inc / bittensor-miner-ddos-mitigation-guide

Guide shared by jondurbin #1