dosubot[bot]
commented
8 months ago 🤖
Based on the information you've provided, it seems like the handshake failure error you're experiencing when trying to run a workflow with the flyte-binary chart and traefik as an Ingress controller in your EKS cluster could be due to the fact that gRPC requires HTTP/2, and not all Ingress controllers support HTTP/2 for backend services.
In the provided context, the Ingress configuration for Flyte is set up to handle both HTTP and gRPC routes. However, not all Ingress controllers can handle both HTTP 1 and gRPC with a single Ingress because gRPC can only be enabled on the Ingress object, not on backend services. This is mentioned in the comments of the provided context:
# Certain ingress controllers like nginx cannot serve HTTP 1 and GRPC with a single ingress because GRPC can only
# enabled on the ingress object, not on backend services (GRPC annotation is set on the ingress, not on the services).Traefik, the Ingress controller you're using, supports HTTP/2, but it needs to be enabled explicitly. If it's not enabled, you might experience handshake failure errors when trying to run gRPC workflows.
To resolve this issue, you need to ensure that HTTP/2 is enabled in your Traefik configuration. If you're using Traefik v2.x, you can enable HTTP/2 by adding the following to your Traefik configuration:
entryPoints:
web:
address: ":80"
http:
http2: truePlease adjust the configuration according to your setup. If you're using a different version of Traefik or have a different setup, the configuration might be different. Please refer to the Traefik documentation for more details.
Regarding the insecureSkipVerify configuration in the Flyte CLI, it is a boolean flag that controls whether a client verifies the server's certificate chain and host name during a gRPC call. If insecureSkipVerify is set to true, the client will not verify the server's certificate chain and host name, which can be insecure and is not recommended for production use cases. This configuration is particularly useful in development or testing environments where you might be using self-signed certificates. In such cases, you can set insecureSkipVerify to true to bypass the SSL certificate verification.
I hope this helps! If you have any other questions or need further clarification, feel free to ask.
Sources
- [BUG] pyflyte run --remote ignores insecureSkipVerify
- charts/flyte-core/templates/common/ingress.yaml
- flyteidl/clients/go/admin/config_flags.go
davidmirror-ops
ldunkum
flixr
Jeinhaus
Describe the bug
We're using traefik as an Ingress controller in our EKS cluster and are deploying the flyte-binary chart. The flyte-binary-grpc service has the annotation
traefik.ingress.kubernetes.io/service.serversscheme: h2c, which according to traefik docs should be fine to serve GRPC traffic.Testing the GRPC endpoints with curl works fine, e.g.:
curl -v -X POST --http2 'https://flyte.example.com/grpc.health.v1.Health' -d "" -H 'Content-Type: application/grpc' -H 'Accept: application/grpc'Trying to run a workflow fails with the following error message:
However, I can create a project from the CLI:
Expected behavior
I'd expect the CLI calls to work without error.
Additional context to reproduce
Screenshots
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?