Possible DDOS issue

[cybermancerr]

My pool was crashing due to below logs and errors. I ended blocking 183.240.196.0/24 network and crashing stopped.

{"level":"info","ts":1530161899.1485713,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"ac6a88db-4330-42b1-ba60-adf7f07df844","ip":"183.240.196.149","uri":"/burst?requestType=getMiningInfo","user-agent":""} 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 160ms 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 5ms 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 5ms {"level":"warn","ts":1530161899.389737,"caller":"pool/pool.go:191","msg":"Miner submitted on invalid height","requestId":"7414b1f4-86d8-46e3-b20b-30852a127f7d","got":506188,"expected":506189} 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 5ms {"level":"warn","ts":1530161899.3954515,"caller":"pool/pool.go:191","msg":"Miner submitted on invalid height","requestId":"99e0ccec-5739-4045-a34f-88d54264cfa3","got":506188,"expected":506189} 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 5ms 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 5ms 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 10ms 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 20ms 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 40ms {"level":"info","ts":1530161899.4665709,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"29495a67-16c6-4bd1-abd4-ca046de9f4f9","accountID":7909028307161998093,"nonce":262724603} {"level":"info","ts":1530161899.4666092,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"29495a67-16c6-4bd1-abd4-ca046de9f4f9"} 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 80ms {"level":"info","ts":1530161899.5110872,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"c5a1c192-fdff-45a3-b84c-dc51d392071e","accountID":6336676392124779714,"nonce":3170787038231368992} {"level":"info","ts":1530161899.5111887,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"c5a1c192-fdff-45a3-b84c-dc51d392071e"} 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 160ms {"level":"info","ts":1530161899.5765386,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"7de098ee-60fd-450f-b843-b0339da082b4","accountID":6336676392124779714,"nonce":3644146842904536106} {"level":"info","ts":1530161899.5765886,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"7de098ee-60fd-450f-b843-b0339da082b4"} 2018/06/27 18:58:19 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 320ms {"level":"info","ts":1530161899.9060771,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"3f82cf27-b1fc-438b-8677-dfce871e543f","ip":"183.240.196.144","uri":"/burst?requestType=getMiningInfo","user-agent":""} 2018/06/27 18:58:20 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 640ms {"level":"info","ts":1530161900.0484722,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"7f02e768-c03c-47e0-8980-74fdbb27a385","ip":"183.240.196.147","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161900.2119117,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"7b1b21f5-60a2-4d1f-9641-6edb9ea56d00","accountID":6336676392124779714,"nonce":8132983812972275705} {"level":"info","ts":1530161900.2119567,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"7b1b21f5-60a2-4d1f-9641-6edb9ea56d00"} {"level":"info","ts":1530161900.2439551,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"e59c29f3-e29b-43ea-98fb-a04873d571df","accountID":6336676392124779714,"nonce":13020253998469113399} {"level":"info","ts":1530161900.2440057,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"e59c29f3-e29b-43ea-98fb-a04873d571df"} {"level":"info","ts":1530161900.2490346,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"8afb15c6-55ac-476d-bdd7-e9ad02f0a380","ip":"183.240.196.148","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161900.401606,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"2236fb0a-e980-4274-add0-5472317555bb","ip":"183.240.196.53","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161900.4829342,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"130245ba-3e3f-4b21-92d5-512b7662e868","accountID":6336676392124779714,"nonce":3644146842922820628} {"level":"info","ts":1530161900.482976,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"130245ba-3e3f-4b21-92d5-512b7662e868"} {"level":"info","ts":1530161900.569727,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"b1f71f94-e89d-462a-b295-e7e00f773940","ip":"183.240.196.53","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161900.5735183,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"d0df95b8-80ee-44f8-8aa7-7aa9808082eb","ip":"183.240.196.144","uri":"/burst?requestType=getMiningInfo","user-agent":""} 2018/06/27 18:58:20 http: Accept error: accept tcp [::]:8124: accept4: too many open files; retrying in 1s {"level":"info","ts":1530161900.7024949,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"19d6bcee-87ca-44eb-87ad-78ea63c31784","ip":"","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161900.9001665,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"7c00e2de-7fcb-4edc-95cd-6da74c04fb80","ip":"183.240.196.144","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161901.0500207,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"4252cc09-5566-44fc-9efa-e19f3f728f25","ip":"183.240.196.151","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161901.0588717,"caller":"pool/pool.go:218","msg":"processing formal valid request","requestId":"97a02421-8d3e-49fe-a22d-14b79ee8efb5","accountID":6336676392124779714,"nonce":8555247405341977372} {"level":"info","ts":1530161901.0589318,"caller":"pool/pool.go:229","msg":"valid reward recipient","requestId":"97a02421-8d3e-49fe-a22d-14b79ee8efb5"} {"level":"info","ts":1530161901.227112,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"49776742-511a-4e70-be8b-9342c28065b6","ip":"183.240.196.60","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161901.2446516,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"10995847-8b76-4fe7-a0ec-043037d322b1","ip":"183.240.196.52","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161901.34792,"caller":"pool/pool.go:278","msg":"rate limit exceeded","requestId":"de1cc5d9-773b-4dcf-8c6a-c06ae5c9f4a2","ip":"183.240.196.52","uri":"/burst?requestType=getMiningInfo","user-agent":""} {"level":"info","ts":1530161901.634269,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"97a02421-8d3e-49fe-a22d-14b79ee8efb5","deadline":12642091} {"level":"info","ts":1530161901.634329,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"e59c29f3-e29b-43ea-98fb-a04873d571df","deadline":5968675} {"level":"info","ts":1530161901.6343536,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"7b1b21f5-60a2-4d1f-9641-6edb9ea56d00","deadline":9972035} {"level":"info","ts":1530161901.634399,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"130245ba-3e3f-4b21-92d5-512b7662e868","deadline":18321496} {"level":"info","ts":1530161901.6342852,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"7de098ee-60fd-450f-b843-b0339da082b4","deadline":21544729} {"level":"info","ts":1530161901.6343055,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"29495a67-16c6-4bd1-abd4-ca046de9f4f9","deadline":23382541} {"level":"info","ts":1530161901.6343179,"caller":"pool/pool.go:253","msg":"valid deadline","requestId":"c5a1c192-fdff-45a3-b84c-dc51d392071e","deadline":25873730} {"level":"info","ts":1530161901.639479,"caller":"pool/pool.go:101","msg":"new best deadline","deadline":12642091} panic: dial tcp 127.0.0.1:3306: socket: too many open files

goroutine 71 [running]: github.com/jmoiron/sqlx.MustExec(0xb123a0, 0xc4202a2060, 0xabe985, 0xb6, 0xc4215bae20, 0x2, 0x2, 0x0, 0xc420349c38) /home/burst/pool/goburstpool/vendor/src/github.com/jmoiron/sqlx/sqlx.go:720 +0xba github.com/jmoiron/sqlx.(DB).MustExec(0xc4202a2060, 0xabe985, 0xb6, 0xc4215bae20, 0x2, 0x2, 0xc42143c000, 0xc421b17a40) /home/burst/pool/goburstpool/vendor/src/github.com/jmoiron/sqlx/sqlx.go:370 +0x70 modelx.(Modelx).UpdateBestSubmission(0xc420254c60, 0x57f0659b01338cc2, 0x7b94d) /home/burst/pool/goburstpool/src/modelx/modelx.go:706 +0x10d pool.(*Pool).forge(0xc42026bef0, 0x7b943, 0x18aed, 0xea5, 0xc4202b4100, 0x40, 0xc4202b4140, 0x20, 0x40, 0x0, ...) /home/burst/pool/goburstpool/src/pool/pool.go:103 +0x528 created by pool.NewPool /home/burst/pool/goburstpool/src/pool/pool.go:57 +0x2a5

[spebern]

The rate limiter takes care of too many requests from the same ip. At the same time you should to increase the limit of open file descriptors: https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

[cybermancerr]

I set crontab -e to: @reboot sleep 10 && cd /home/burst/burstcoin/ && ./burst.sh @reboot sleep 60 && cd /home/burst/pool/testpool/ && ./persist.sh

Then put this into persist.sh:

!/bin/bash

include this boilerplate

function jumpto { label=$1 cmd=$(sed -n "/$label:/{:a;n;p;ba};" $0 | grep -v ':$') eval "$cmd" exit }

start=${1:-"start"}

jumpto $start

start: ulimit -Hn 1048576 ulimit -Sn 1048576 cd /home/burst/pool/goburstpool/ ./goburstpool cd /home/burst/pool/testpool/ jumpto start

Will be monitoring process output with tail -f /proc/1850/fd/1 for 24 hours to make sure all is well and will post results here.

Thanks for the help.

[cybermancerr]

Pool is stable.

Thank you, Khomitchr

[spebern]

you are welcome