Fasthttp panics

[AkramHssaini]

When sending multiple requests with multiple threads over a long period, fasthttp crashes. I'm sending multiple requests to a server with a 'keep-alive' header, using around 100 goroutines. The program runs well for the first 30-40 minutes, but then it suddenly crashes with a stack call. I've been trying to understand it but failed. I've tried changing 'MaxConnsPerHost' and 'MaxIdleConnDuration,' but it only resulted in a bit more time before crashing. The error handling is also implemented, but fasthttp just panics, and the server is capable of responding and handling this amount of requests. I'm not able to understand what causes the issue, so I'd be glad if anyone can help me.

My configuration:

client := &fasthttp.Client{
  Dial:                (&fasthttp.TCPDialer{Concurrency: 4096}).Dial,
  MaxConnsPerHost:     int(InitialSize) * 2,
  MaxIdleConnDuration: 16 * time.Second,
  ReadTimeout:         15 * time.Second,
  WriteTimeout:        15 * time.Second,
  // ReadBufferSize:      8192,
  // WriteBufferSize:     8192,
  ConnPoolStrategy: fasthttp.LIFO,
}
client.Dial = fasthttpproxy.FasthttpHTTPDialerTimeout("user:pass@ip:port", time.Second*15)

The Stackcall

runtime.gopark(0x0?, 0xc0bae6f198?, 0x48?, 0xf2?, 0xc0bae6f1c8?)
        C:/Program Files/Go/src/runtime/proc.go:398 +0xce fp=0xc0b3b0f0e8 sp=0xc0b3b0f0c8 pc=0xb49eee
runtime.netpollblock(0xc167bfb97e63c66b?, 0xb149a6?, 0x0?)
        C:/Program Files/Go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0b3b0f120 sp=0xc0b3b0f0e8 pc=0xb41077
internal/poll.runtime_pollWait(0x200c8a9a1a0, 0x72)
        C:/Program Files/Go/src/runtime/netpoll.go:343 +0x85 fp=0xc0b3b0f140 sp=0xc0b3b0f120 pc=0xb70fc5
internal/poll.(*pollDesc).wait(0xdc56cae3d3c88a73?, 0xc0b3b0f250?, 0x0)
        C:/Program Files/Go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0b3b0f168 sp=0xc0b3b0f140 pc=0xbc7de7
internal/poll.execIO(0xc0bae6f198, 0xe1d1e0)
        C:/Program Files/Go/src/internal/poll/fd_windows.go:175 +0xe6 fp=0xc0b3b0f1d8 sp=0xc0b3b0f168 pc=0xbc8566
internal/poll.(*FD).Read(0xc0bae6f180, {0xc0c4efb500, 0x1500, 0x1500})
        C:/Program Files/Go/src/internal/poll/fd_windows.go:436 +0x2b1 fp=0xc0b3b0f280 sp=0xc0b3b0f1d8 pc=0xbc9211
net.(*netFD).Read(0xc0bae6f180, {0xc0c4efb500?, 0xc0005041a0?, 0x200000003?})
        C:/Program Files/Go/src/net/fd_posix.go:55 +0x25 fp=0xc0b3b0f2c8 sp=0xc0b3b0f280 pc=0xc399a5
net.(*conn).Read(0xc0c6cea860, {0xc0c4efb500?, 0xc0be6c82c8?, 0x1b85910600000000?})
        C:/Program Files/Go/src/net/net.go:179 +0x45 fp=0xc0b3b0f310 sp=0xc0b3b0f2c8 pc=0xc45805
net.(*TCPConn).Read(0xc0b3b0f3a8?, {0xc0c4efb500?, 0xc0d5d72f18?, 0x18?})
        <autogenerated>:1 +0x25 fp=0xc0b3b0f340 sp=0xc0b3b0f310 pc=0xc4ec25
crypto/tls.(*atLeastReader).Read(0xc0d5d72f18, {0xc0c4efb500?, 0xc0d5d72f18?, 0x0?})
        C:/Program Files/Go/src/crypto/tls/conn.go:805 +0x3b fp=0xc0b3b0f388 sp=0xc0b3b0f340 pc=0xd036db
bytes.(*Buffer).ReadFrom(0xc0c7905ea8, {0xe690a0, 0xc0d5d72f18})
        C:/Program Files/Go/src/bytes/buffer.go:211 +0x98 fp=0xc0b3b0f3e0 sp=0xc0b3b0f388 pc=0xbe0fd8
crypto/tls.(*Conn).readFromUntil(0xc0c7905c00, {0x200fedfc938?, 0xc0c6cea860}, 0xbb0fd4?)
        C:/Program Files/Go/src/crypto/tls/conn.go:827 +0xde fp=0xc0b3b0f420 sp=0xc0b3b0f3e0 pc=0xd038be
crypto/tls.(*Conn).readRecordOrCCS(0xc0c7905c00, 0x0)
        C:/Program Files/Go/src/crypto/tls/conn.go:625 +0x250 fp=0xc0b3b0f7c0 sp=0xc0b3b0f420 pc=0xd00e90
crypto/tls.(*Conn).readRecord(...)
        C:/Program Files/Go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc0c7905c00, {0xc0d1569000, 0x1000, 0xd4bfef?})
        C:/Program Files/Go/src/crypto/tls/conn.go:1369 +0x158 fp=0xc0b3b0f830 sp=0xc0b3b0f7c0 pc=0xd07158
bufio.(*Reader).fill(0xc0c16ad3e0)
        C:/Program Files/Go/src/bufio/bufio.go:113 +0x103 fp=0xc0b3b0f868 sp=0xc0b3b0f830 pc=0xc1fc03
bufio.(*Reader).Peek(0xc0c16ad3e0, 0x1)
        C:/Program Files/Go/src/bufio/bufio.go:151 +0x53 fp=0xc0b3b0f888 sp=0xc0b3b0f868 pc=0xc1fd33
github.com/valyala/fasthttp.(*ResponseHeader).tryRead(0xc0b82dc540, 0xc0b3b0f9b0?, 0x1)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/header.go:2027 +0x4c fp=0xc0b3b0f930 sp=0xc0b3b0f888 pc=0xd533cc
github.com/valyala/fasthttp.(*ResponseHeader).Read(0xc0b82dc540?, 0xc0c16ad3e0)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/header.go:2013 +0x3d fp=0xc0b3b0f970 sp=0xc0b3b0f930 pc=0xd532fd
github.com/valyala/fasthttp.(*Response).ReadLimitBody(0xc0b82dc540, 0xe6b850?, 0xc0c7905c00?)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/http.go:1398 +0x38 fp=0xc0b3b0f9c0 sp=0xc0b3b0f970 pc=0xd5ed98
github.com/valyala/fasthttp.(*transport).RoundTrip(0xc0b82dc540?, 0xc0b51ff380, 0xc0b35f8a80, 0xc0b82dc540)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/client.go:2922 +0x575 fp=0xc0b3b0fab0 sp=0xc0b3b0f9c0 pc=0xd4dc35
github.com/valyala/fasthttp.(*HostClient).doNonNilReqResp(0xc0b51ff380, 0xc0b35f8a80, 0xc0b82dc540)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/client.go:1373 +0x336 fp=0xc0b3b0fb58 sp=0xc0b3b0fab0 pc=0xd49ab6
github.com/valyala/fasthttp.(*HostClient).do(0xb259eb?, 0x1062140?, 0xbf16eb?)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/client.go:1323 +0xae fp=0xc0b3b0fbc0 sp=0xc0b3b0fb58 pc=0xd4966e
github.com/valyala/fasthttp.(*HostClient).Do(0xc0b51ff380, 0xc0b35f8a80, 0xc0b2f349a0?)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/client.go:1268 +0x173 fp=0xc0b3b0fc58 sp=0xc0b3b0fbc0 pc=0xd49373
github.com/valyala/fasthttp.(*Client).Do(0xc0b74d83c0, 0xc0b35f8a80, 0x162?)
        C:/Users/Akram/go/pkg/mod/github.com/valyala/fasthttp@v1.51.0/client.go:551 +0x5fb fp=0xc0b3b0fd48 sp=0xc0b3b0fc58 pc=0xd48a5b

[erikdubbelboer]

What is the actual panic message? And which version of Go and fasthttp are you using?

[AkramHssaini]

The stack call is all I get there somehow isn’t an actual message. Go and Fasthttp are all up to date

[erikdubbelboer]

Are you sure you are on the latest Go 1.21.6? src/runtime/proc.go:398 is not a line that can panic on that version. There should always be a message before the panic. Can you share the full output of your program?

Since the panic is so deep into the Go runtime I don't think this has anything to do with fasthttp. It seems to me like some kind of memory corruption. Are you maybe using any cgo libraries? Could it be that your system memory is broken?

[AkramHssaini]

You were right; the issue wasn't caused by fasthttp. It was a different package that caused the crash. Thank you a lot for the help.