Closed sjke closed 6 years ago
ulimit -Hn 1000000; ulimit -n 1000000
before starting the service. you can also find systemd service variables that will setup this for you
@kirillDanshin thx, but this isn't a fix for the issue (by increasing the limits, I'm just "delaying the inevitable"). I have to understand the reason: "Why it happens" and "How can I prevent it"
I figured out the problem in TCPListener
.
Close!
@sjke to be precise, you should check that your listener is correctly closing connections (if you use custom listener), decrease TTL for your keep alive connections (or disable keep-alive) and set those limits to a max number of opened connections at once.
basically, you just found a common issue, when you got 4096 opened connections and tried to open 4097th, but it can't work due to ulimits. each socket (including client-server connections) in linux are files, so to serve more connections, you'll need higher limits and always correctly handle connection close.
@sjke Could you explain us exactly your error? please
@savsgio Did you find a solution?
No!, but @kirillDanshin has the reason, I think it's very probably the error is by the listener.
@savsgio thanks for answer, but he not explained how did... sad!
I think it too! :confused:
@rof20004 are you having an issue with too many open files
as well?
@erikdubbelboer Sometimes yes, sometimes no.
My ulimit is 100000(one hundred thousand).
My test is with vegeta.
echo "GET http://localhost:8080" | vegeta attack -rate=20000 -duration=30s | vegeta report
My code is this:
package main
import (
"fmt"
"log"
"github.com/buaazp/fasthttprouter"
"github.com/valyala/fasthttp"
)
func Index(ctx *fasthttp.RequestCtx) {
fmt.Fprint(ctx, "Welcome!\n")
}
func main() {
router := fasthttprouter.New()
router.GET("/", Index)
log.Fatal(fasthttp.ListenAndServe(":8080", router.Handler))
}
Running this as root on my macbook I only get:
$ echo "GET http://localhost:8080" | sudo vegeta attack -rate=20000 -duration=30s | vegeta report
Password:
Requests [total, rate] 600000, 20000.55
Duration [total, attack, wait] 29.999536238s, 29.999175s, 361.238µs
Latencies [mean, 50, 95, 99, max] 256.769µs, 136.854µs, 378.63µs, 1.195158ms, 81.91395ms
Bytes In [total, mean] 5396796, 8.99
Bytes Out [total, mean] 0, 0.00
Success [ratio] 99.94%
Status Codes [code:count] 0:356 200:599644
Error Set:
Get http://localhost:8080: dial tcp: lookup localhost: no such host
The only error there has nothing to do with fasthttp. Could it be that you're running your test as a user that runs out of file descriptors?
@erikdubbelboer Thanks, maybe my user had a limit or I not setted ulimit correctly, I tested yesterday after set ulimit to one hundred again and no errors happens.
@rof20004 I recomend you to use https://github.com/fasthttp/router also. The router is maintained by fasthttp community.
@savsgio Thanks, I will use it.
I have one very unpleasant issue with the fasthttp : There is a "highly loaded" server that stops processing requests after 5-10 minutes (about 5000 requests per minute) with error:
Temporary error when accepting new connections: accept tcp4 IP:PORT: accept: too many open files
Tried 2 ways to start the (with standard
ListenAndServe
and with custom Server options (disabled Keepalive, custom timeouts and closing connection on handers) but isn't works.