Slow client is leading to lost connection

[xpr0ger]

I have two issues. The major one is connection lost. I have a client that consumes a big text stream at low speed. For example 100Kbps. After some time the client loses its connection to the server. Usually, it happens after that the client has downloaded 8Mb.

On server side is nothing, no any logs or exceptions or something else. I just can't get any debug information for this case an this is the second issue. Even in the trace mode, netty does not make any output for connections or others actions and ktor also, except startup log.

In opposite to netty, with jetty everything is ok except it does not have flush implementation and this is an issue to me.

I have couple questions: How can I get debugging information from netty? And how I can eliminate disconnection on a slow client?

I tried to change write and idle timeout but no effect.

NettyApplicationEngine(kodein.instance()) {
            configureBootstrap = {
                ServerBootstrap().apply {
                    childOption(ChannelOption.SO_KEEPALIVE, true)
                    childHandler(IdleStateHandler(300, 300, 300))
                    childHandler(WriteTimeoutHandler(300, TimeUnit.SECONDS))
                }
            }
        }

Ktor version 0.9.0

[cy6erGn0m]

with jetty everything is ok except it does not have flush implementation

what do you mean?

ChannelOption.SO_KEEPALIVE, true

this wouldn't help as the default TCP keepalive period is usually very long (usually hours)

childHandler(WriteTimeoutHandler(300, TimeUnit.SECONDS))

it is already installed and should help, wonder why it doesn't work. looking into

[xpr0ger]

with jetty everything is ok except it does not have flush implementation

what do you mean?

Ok, I'm was wrong, jetty has the flush implementation but it also has a quite nasty issue. It can't flush only headers, you have to write something to the body before you call flush.

Examples:

You wont see the status code 200 after flush.

call.respondWrite(status = HttpStatusCode.OK) {
    flush()
    ... do something
}

But if you'll do this:

call.respondWrite(status = HttpStatusCode.OK) {
    write(" ") // you have to write something not an empty string
    flush()
    ... do something
}

In this case, you'll see status code immediately after flush

Can I somehow avoid it?

[cy6erGn0m]

I believe flush() should force response body in both cases. How could we flush empty character without sending status? It is impossible in HTTP.

[xpr0ger]

We have miss understood. I want to force the headers before the body. For example, for long polling implementation, I should send status code right after a connection was established even in that case when I don't have any of data to send. But jetty won't do that.

I think in first case headers should be sent even if the body is empty.

[cy6erGn0m]

I agree, will fix it

[xpr0ger]

I agree, will fix it

Thank you.

And what about the issue with a slow client, could you please give me some advice how can I solve that or where I should start investigating this problem?

[cy6erGn0m]

In netty there is a write timeout specified of 10 seconds (not configurable for now, will add it). Unfortunately netty's ktor engine doesn't log such errors for now so network connection is silently closed. The best advice is to wait for upcoming 0.9.1-alpha-6 that is going to be released soon once the build becomes green.

[cy6erGn0m]

Partially fixed in 0.9.1-alpha-6: now jetty should flush status and headers while netty should close connection on timeout, but no timeout configuration yet and no appropriate logging

[cy6erGn0m]

Related to #538

[frost13it]

The issue is still there (Ktor 1.2.3). An attempt to transfer 10 MB to a slow (100 kB/s) client always ends up with closed connection after 40 seconds when default settings are used.

Steps to reproduce:

1. Write a handler that responds with 10 MB byte array:

   embeddedServer(Netty) {
   routing {
       get("/") {
           context.respond(ByteArray(10_048_576))
       }
   }
   }

2. Run wget -t1 --limit-rate=100k -O /dev/null localhost

Expected behavior: Ktor should not close connection when transmission is in progress.

Actual behavior: Connection closed at byte 4147317.

[oleg-larshin]

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.

[vvalouch]

I can confirm the reproducibility even with Ktor(server) 1.4.0 and Ktor 1.4.1 and curl as client.

[ademarizu]

Any news on this one? I'm facing the same problem. Ktor: 1.5.4 Kotlin: 1.4.10