Post es http err:http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

[franklwel1990]

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

github.com/olivere/elastic/v7 v7.0.22

Please describe the expected behavior

Post "https://aaaes.amazonaws.com/statistics_%2A/_search": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error server/ChessCronApi/es.(*FundData).FundMoneySearch

Please describe the actual behavior

Online requests for es addresses often report errors on stage

Any steps to reproduce the behavior?

no

[olivere]

I need to further investigate. There's an ancient issue at https://github.com/golang/go/issues/25009 which describes this bug but is closed at least after Go 1.12. I've never seen this message after years of running in production.

[GrahamWalters]

I'm also seeing this error, but we're using a GET request

Get "https://xyz.us-east-1.es.amazonaws.com/xyz/_doc/1157": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

github.com/olivere/elastic/v6 v6.2.16 go version go1.15 linux/amd64

[fengyuxx]

Same error in bulk write request. It looks like we are all using aws?

Post https://xxxxxxxxx.us-west-2.es.amazonaws.com/_bulk: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error.

[rbUUbr]

@olivere seems like usage of go 1.15 version should help to avoid this problem according to this thread: https://github.com/golang/go/issues/32441

[Atply234]

Any updates on this ? Im facing the similar issue and also upgrade of go to 1.15 did not help.

[juhovuori]

I'm also seeing this regularly on go 1.16.

[kahluw2]

I am seeing this in go 1.14 and 1.15 both. Any updates on this?

[olivere]

It looks like this is the source of the error. I'm looking into this in the next release.

It should be a simple fix.

P.S.: Go 1.14 is no longer supported, and 1.15 is on the edge (once 1.17 comes out in August 2021). I think it's a good idea to update anyway—even if it doesn't fix your issue.

[Hhhha]

Maybe it's http2. You can set http.Transport.ForceAttemptHTTP2 to false.

[olivere]

Anybody can confirm this still happens in the supported Go versions, i.e. Go 1.16 and Go 1.17 (at the time I'm writing this)?

[vladiacob]

@olivere It's still happening pretty often. We didn't have that issue until we migrated from elasticsearch to AWS OpenSearch.

[olivere]

@olivere It's still happening pretty often. We didn't have that issue until we migrated from elasticsearch to AWS OpenSearch.

@vladiacob May I ask which Go version you're using?

[vladiacob]

@olivere 1.17.1 is the version

[vladiacob]

Do you have any idea how we can solve the issue? Is happening really often.

[jaakkotuosa]

Happening multiple times a day for me on go 1.17.1 + AWS OpenSearch (Elasticsearch 7.9, connections from ECS, through VPN)

[hexa00]

We experienced this issue on go 1.16 also and worked-around it by building with: go build -tags nethttpomithttp2

[ceverus]

I am seeing this in go 1.14.

[pgvishnuram]

Yes its observed in aws opensearch and aws elasticsearch.

[pgvishnuram]

Issue can be reproduced when you try to write huge data or simultaneous data writes to elastic search

[nblizard]

We were not running into this issue when using only http, but once we started using https, this issue cropped up for us. This appeared to occur very intermittently, but it was reproducible and occurred when using basic auth or AWS IAM Auth4.

@hexa00 's solution above did indeed work for us, but due to our codebase, we were hesitant to to use a compiler directive to affect all httpClients.

After some research, I believe I have a solution. I believe the piece of info we are looking for is outlined here: https://pkg.go.dev/net/http#pkg-overview. "Programs that must disable HTTP/2 can do so by setting Transport.TLSNextProto (for clients) or Server.TLSNextProto (for servers) to a non-nil, empty map. Alternatively, the following GODEBUG environment variables are currently supported."

Our full solution below appears (thus far) to work for us:

httpClient := &http.Client{
    Transport: &http.Transport{
        TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
    },
}

signingClient := elasticaws.NewV4SigningClientWithHTTPClient(credentials.NewStaticCredentials(env.ESKeyID, env.ESSecretKey, ""), env.AWSRegion, httpClient)
options := []elastic.ClientOptionFunc{elastic.SetURL(env.ESURL), elastic.SetHttpClient(signingClient), elastic.SetHealthcheck(false), elastic.SetSniff(false)}

client := elastic.NewClient(options...)

[JoydS]

Hello guys, So basically we need to disable http2 to fix this problem ? On my side, it's not on a post request but a get request. Would be great to have a fix on this, thanks guys