Getting TooLongFrameException HTTP header is larger than 8192 bytes

[mfirry]

I'm using sttp 3.0.0 with async-http-client-backend-cats. My code looks like this:

      AsyncHttpClientCatsBackend.resource[IO]().use { implicit backend =>
        basicRequest
          .get(uri"${myUrl}")
          .response(asString)
          .send(backend)
      }

It might not be sttp, but I just would like to know if it's a known thing. I tried putting something like javaOptions ++= Seq("-Dhttp.netty.maxHeaderSize=16384"), to no avail.

[error] (run-main-13) io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.
[error] io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.

[adamw]

Hm interesting. Is the myUrl really long?

[adamw]

If the URL is very long, maybe you need to specify -Dhttp.netty.maxInitialLineLength, as suggested here: https://github.com/playframework/playframework/issues/5036

[mfirry]

It actually is a bit. What makes it even longer is a parameter which can contain a comma-separated value. (It's not something I can change, unfortunately), It looks like this:

https://www.something.it/ajax/rest/model/com/something/cube/commerce/inventory/Inventory/getStoreAvailability?storeIds=0070060800608,0070049300493,0070060900609,0070045700457,0070081600816,0070054500545,0070053400534,0070231402314,0070081000810,0070070600706&skuId=2962862&modelId=8575940&displayStoreDetails=false

(I had to anonymize it a little bit)

[mfirry]

I've put

javaOptions ++= Seq(
  "-Dhttp.netty.maxHeaderSize=16384",
  "-Dhttp.netty.maxInitialLineLength=16384"
)

But still same error both in sbt run and if I run it as java -jar ... with the uberjar.

[mfirry]

Even when running java -Dhttp.netty.maxHeaderSize=16384 -jar target/... I have the same error.

[adamw]

I've been trying to reproduce this but failed so far. Here's what I've got:

object Test extends App {
  implicit val contextShift: ContextShift[IO] = IO.contextShift(ExecutionContext.global)

  val r = AsyncHttpClientCatsBackend
    .resource[IO]()
    .use { backend =>
      val query = List.fill(3000)("0070060800608").mkString(",")
      val myUrl = s"""https://httpbin.org/get?storeIds=$query"""
      println(s"URL length: ${myUrl.length}")
      basicRequest
        .get(uri"${myUrl}")
        .response(asStringAlways)
        .send(backend)
    }
    .unsafeRunSync()

  println(s"Response: ${r.code} ${r.statusText}")
}

In response I get:

URL length: 42032
Response: 414 Request-URI Too Large

so the server rejects the request, but the request itself is sent

[adamw]

Try:

val config = new DefaultAsyncHttpClientConfig.Builder().setHttpClientCodecMaxHeaderSize(16384).build()

and later:

AsyncHttpClientCatsBackend
    .resourceUsingConfig[IO](config)
    .use { backend => ... }

[mfirry]

This works!

[mfirry]

It also works setting it like this:

javaOptions ++= Seq("-Dorg.asynchttpclient.httpClientCodecMaxHeaderSize=16384")