Error reading preface error in the tekton results api controller logs in the Openshift with enabled http/2

[AndrienkoAleksandr]

Expected Behavior

Tekton results api controller logs should be clean.

Actual Behavior

Tetkon result api controller logs flooded with errors:

http2: server: error reading preface from client 10.x.y.z:44444: read tcp 10.v.z.n:8443->10.x.y.z:44444: read: connection reset by peer

Steps to Reproduce the Problem

Note: issue is reproducible in the Openshift 4 with enabled http/2

1. Deploy tekton results api in the Openshift.
2. Create http route with reencrypt tls termination.
3. Take a look logs of the Tekton results controller.

Known workarounds:

1. Disable http/2. To disable http/2 for tekton-results api controller pod you can use env:

… env:

• name: GODEBUG value: http2server=0 …

But this workaround can be not desirable.

3. Disable verbose logging

This workaround described in related article https://access.redhat.com/solutions/6999058, but looks like this workaround doesn't work. Because Tekton results controller doesn't use GODEBUG=http2debug=1 or GODEBUG=http2debug=1. Env variable GODEBUG is unset at all.

Additional Info

• Kubernetes version:

Reproduced on the Openshift 4 with enabled http/2.

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

[adambkaplan]

@sayan-biswas @avinal if I recall, you ran into the opposite issue trying to run the Results e2e suite on OpenShift when the apiserver is exposed via a Route? Do we know if gRPC connections require http2 to be enabled end to end?

[sayan-biswas]

@adambkaplan There are two issues with the e2e.

• Since running the e2e script in an openshift requires a route and it also access results service through gRPC client to run the gRPC test cases, the http2 needs to be enabled.
• A test openshift cluster doesn't get a valid TLS ertificate and currently the insecureVerify is not implemented for the clients in the e2e code.

For this issue, it's not related to ours. I'll check why this is happening. And we don't need http2 to be enabled end to end unless the tkn-results cli is used. Since this configuration is at the ingress level, internal traffic will not be affected. But for the pod it needs to be enabled I guess.

[sayan-biswas]

@adambkaplan This is not an issue with tekton results. A feature in haproxy open TCP connection for health check and closes it without handshake. Discussion can be found here.. https://bugzilla.redhat.com/show_bug.cgi?id=2111670

There is a workaround though. Separating the port used in the Openshiftt route in a different doesn't produce this error. In this case we have metrics ports and API server ports in the same service. Since this is an Openshift specific issue, do you think it would be a good idea to provide this workaround here? Ideally this should be an Openshift bug and is already registered in their tracker.