🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka® via declaratively defined, stateless APIs.
Describe the bug
In case zilla is set up as an http proxy between an nginx h2 server and a curl http/1.1 client, if response validation is enabled, and the response is invalid, zilla displays an erratic behavior as described below.
To Reproduce
Steps to reproduce the behavior:
Prerequisites:
we need to enable http response validation (as of now this is a PR on review that's not merged to develop yet)
One of these cases are happening, ususally in this order:
1) expected case, curl prints:
curl: (18) transfer closed with 5 bytes remaining to read
2) curl hangs, must be stopped with ^C
3) curl displays an error message:
* Excess found in a read ...
* Closing connection
4) zilla crashes, with an AssertionError with a stack trace like this:
Caused by: java.lang.AssertionError
at io.aklivity.zilla.runtime.binding.http/io.aklivity.zilla.runtime.binding.http.internal.stream.HttpClientFactory$HttpExchange.onResponseWindow(HttpClientFactory.java:4941)
at io.aklivity.zilla.runtime.binding.http/io.aklivity.zilla.runtime.binding.http.internal.stream.HttpClientFactory$HttpExchange.onApplication(HttpClientFactory.java:4569)
at io.aklivity.zilla.runtime.binding.http/io.aklivity.zilla.runtime.binding.http.internal.stream.HttpClientFactory$HttpClientPool.lambda$newStream$2(HttpClientFactory.java:2086)
at io.aklivity.zilla.runtime.engine/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleReadReply(EngineWorker.java:1276)
Expected behavior
The curl command pointed out above should consistently result in the expected case (1) as described above.
Additional context
We have investigated this issue with @akrambek and concluded that this is most likely a flow control bug in the http/1.1 server.
The issue only appears in the case zilla client -> nginx use h2 and curl -> zilla server use http/1.1.
May or may not be related... I bumped into this other issue where the onReponseBegin in the exchange crashed on the highlighted assert where acknowledge was 702 and responseAck was 0.
Describe the bug In case zilla is set up as an http proxy between an nginx h2 server and a curl http/1.1 client, if response validation is enabled, and the response is invalid, zilla displays an erratic behavior as described below.
To Reproduce Steps to reproduce the behavior:
Prerequisites:
develop
yet)IntegerValidator
(this is being fixed on a feature branch, as of now that's not merged todevelop
yet) https://github.com/aklivity/zilla/commit/30a7de0b45f841020d81489603b8c0408b89816dSetup:
42.integer
with a content length != 4 characters in<some-directory>
(to simulate the integer validation invalid case)nginx.conf
in<some-directory>
with h2 enabled (this a defaultnginx.conf
, only with theserver { listen 80 http2; }
block inserted):error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid;
events { worker_connections 1024; }
http { server { listen 80 http2; }
}
docker run -p 8080:80 -v:/usr/share/nginx/html:ro -v /nginx.conf:/etc/nginx/nginx.conf --rm --name nginx nginx
name: example bindings: north_tcp_server: type: tcp kind: server options: host: 0.0.0.0 port: 7114 exit: north_http_server north_http_server: type: http kind: server exit: south_http_client south_http_client: type: http kind: client options: versions:
JAVA_OPTIONS="-ea -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" ./zilla start -v -e -w 1
curl -v http://localhost:7114/42.integer
One of these cases are happening, ususally in this order: 1) expected case, curl prints:
2) curl hangs, must be stopped with ^C 3) curl displays an error message:
4) zilla crashes, with an
AssertionError
with a stack trace like this:Expected behavior The
curl
command pointed out above should consistently result in the expected case (1) as described above.Additional context We have investigated this issue with @akrambek and concluded that this is most likely a flow control bug in the http/1.1 server.
The issue only appears in the case zilla client -> nginx use h2 and curl -> zilla server use http/1.1.