A panic may occur when a large file download

[yinjianping07]

when I downloaded a file of about 9G, i got this 2022/11/04 18:02:45 http: panic serving [::1]:52274: runtime error: slice bounds out of range [:10200547328] with capacity 2147483647 goroutine 6 [running]: net/http.(conn).serve.func1(0x140004ad220) /Users/yinjianping/go/go1.17.11/src/net/http/server.go:1802 +0xdc panic({0x103581060, 0x1408050c000}) /Users/yinjianping/go/go1.17.11/src/runtime/panic.go:1052 +0x2ac github.com/valyala/fasthttp.appendBodyFixedSize(0x14000200120, {0x103fa3e08, 0x0, 0x0}, 0x260000000) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/http.go:2167 +0x2a0 github.com/valyala/fasthttp.readBody(0x14000200120, 0x260000000, 0x0, {0x103fa3e08, 0x0, 0x0}) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/http.go:2086 +0x9c github.com/valyala/fasthttp.(Response).ReadBody(0x14000138000, 0x14000200120, 0x0) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/http.go:1380 +0x1e0 github.com/valyala/fasthttp.(Response).ReadLimitBody(0x14000138000, 0x14000200120, 0x0) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/http.go:1349 +0xe8 github.com/valyala/fasthttp.(HostClient).doNonNilReqResp(0x14000102820, 0x1400013c000, 0x14000138000) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/client.go:1420 +0x808 github.com/valyala/fasthttp.(HostClient).do(0x14000102820, 0x1400013c000, 0x14000138000) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/client.go:1289 +0x90 github.com/valyala/fasthttp.(HostClient).Do(0x14000102820, 0x1400013c000, 0x14000138000) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/client.go:1236 +0xc0 github.com/valyala/fasthttp.(Client).Do(0x1400013e000, 0x1400013c000, 0x14000138000) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/client.go:544 +0x610 github.com/valyala/fasthttp.(Client).DoTimeout(...) /Users/yinjianping/go/pkg/mod/github.com/valyala/fasthttp@v1.41.0/client.go:391 main.indexHandler({0x10364b460, 0x14000132000}, 0x140004ced00) /Users/yinjianping/go/src/baidu/personal-code/yinjianping/test/main.go:130 +0x1ec net/http.HandlerFunc.ServeHTTP(0x103629e20, {0x10364b460, 0x14000132000}, 0x140004ced00) /Users/yinjianping/go/go1.17.11/src/net/http/server.go:2047 +0x40 net/http.(ServeMux).ServeHTTP(0x103f727c0, {0x10364b460, 0x14000132000}, 0x140004ced00) /Users/yinjianping/go/go1.17.11/src/net/http/server.go:2425 +0x18c net/http.serverHandler.ServeHTTP({0x140001f81c0}, {0x10364b460, 0x14000132000}, 0x140004ced00) /Users/yinjianping/go/go1.17.11/src/net/http/server.go:2879 +0x444 net/http.(conn).serve(0x140004ad220, {0x10364f0e0, 0x14000493d10}) /Users/yinjianping/go/go1.17.11/src/net/http/server.go:1930 +0xb6c created by net/http.(*Server).Serve /Users/yinjianping/go/go1.17.11/src/net/http/server.go:3034 +0x4b8

[byene0923]

this use ReadLimitBody， may be you should use streaming

[erikdubbelboer]

fasthttp is not meant for that. Please just use net/http!

Fasthttp is for some really specific cases where performance is super important. Because of this fasthttp always reads the full body into memory. So if you have a 9GB response it's going to use at least 9GB of memory.

[yinjianping07]

fasthttp is not meant for that. Please just use net/http!

Fasthttp is for some really specific cases where performance is super important. Because of this fasthttp always reads the full body into memory. So if you have a 9GB response it's going to use at least 9GB of memory.

My current solution: net/http, thx!

[blacktop]

Is this still the case that fasthttp shouldn't be used for large file downloads? If so are there any pkgs that do speed up these large file downloads similar to how aria2c works?

[erikdubbelboer]

Yes this will always be the case. Large file downloads are bandwidth limited, fasthttp isn't going to speed up anything there.