line / armeria

Your go-to microservice framework for any situation, from the creator of Netty et al. You can build any type of microservice leveraging your favorite technologies, including gRPC, Thrift, Kotlin, Retrofit, Reactive Streams, Spring Boot and Dropwizard.
https://armeria.dev
Apache License 2.0
4.83k stars 921 forks source link

gRPC: service call failing with TLS handshake failed: javax.net.ssl...routines:OPENSSL_internal:WRONG_VERSION_NUMBER #2598

Open RikuVan opened 4 years ago

RikuVan commented 4 years ago

-armeria and armeria-grpc verisons 0.98.7

I have copied the grpc-kotlin example from the repo and tried it with a grpc-web frontend that has worked before behind a proxy and a different backend. I have tried both the tsSelfSigned()and ts(cert, key) but both result in the same exception belore when a call is made from grpc-web. I have also tried with the cors builder as suggested in the armeria grpc docs.

javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1264) ~[netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1225) ~[netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1296) ~[netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1339) ~[netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340) [netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) [netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:253) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.47.Final.jar:4.1.47.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.47.Final.jar:4.1.47.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
[WARN ] c.l.a.s.HttpServerPipelineConfigurator - [id: 0xfc6c9e1e, L:/0:0:0:0:0:0:0:1:8080 - R:/0:0:0:0:0:0:0:1:57328] TLS handshake failed:
javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
trustin commented 4 years ago

Hi! That's interesting. 1) What frontend do you use? 2) What happens if you connect via the proxy server you were using?

RikuVan commented 4 years ago

So I realized now that we have to use a proxy (Envoy) as we want to use grpcwebtext (with streaming calls not just unary) format not grpcweb. We already have quite a lot of typescript generated which works with a basic netty server but we would like some of the extra features of Armeria. So far I get the same error with the proxy.

RikuVan commented 4 years ago

Managed to arrive at a different error at least: [WARN ] c.l.a.s.HttpServerPipelineConfigurator - [id: 0xbe277d56, L:/127.0.0.1:9090 - R:/127.0.0.1:56898] TLS handshake failed: javax.net.ssl.SSLHandshakeException: error:10000418:SSL routines:OPENSSL_internal:TLSV1_ALERT_UNKNOWN_CA

RikuVan commented 4 years ago

this is with

        val key = File("../certs/server.pem")

        return Server.builder()
            .https(port)
            .tls(cert, key)
RikuVan commented 4 years ago

the Envoy proxy to which I added the transport_socket block which I am guessing is where the current problem lies :

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ['*']
                      routes:
                        - match: { prefix: '/' }
                          route:
                            cluster: api_service
                            max_grpc_timeout: 0s
                      cors:
                        allow_origin:
                          - '*'
                        allow_methods: GET, PUT, DELETE, POST, OPTIONS
                        allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                        max_age: '1728000'
                        expose_headers: custom-header-1,grpc-status,grpc-message
                http_filters:
                  - name: envoy.grpc_web
                  - name: envoy.cors
                  - name: envoy.router
  clusters:
    - name: api_service
      connect_timeout: 0.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      hosts: [{ socket_address: { address: dev-backend, port_value: 9090 } }]
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          '@type': type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext
          common_tls_context:
            tls_certificates:
              certificate_chain: { 'filename': 'etc/server.crt' }
              private_key: { 'filename': 'etc/server.pem' }
            validation_context:
              verify_subject_alt_name: [foo]
              trusted_ca:
                filename: /etc/ca.crt
trustin commented 4 years ago

TLSV1_ALERT_UNKNOWN_CA

It sounds like Envoy does not trust the certificate sent by Armeria.

If the certificate is a proper one (e.g. you purchased it, etc), you might want to do one of the following:

Another question is, do you intend to use TLS between Envoy and Armeria? You might just want to use plaintext communication between Envoy and Armeria unless you want to achieve forward secrecy.

RikuVan commented 4 years ago

Yes how to do you use plain text with Armeria, something like .usePlaintext()I suppse. I will try it out. Actually this is a closed system so we don't need ssl yet; we probably will in the future.

trustin commented 4 years ago

When you build a Server with a ServerBuilder, just do not call https(port); use .http(port). TLS is completely optional in Armeria.

RikuVan commented 4 years ago

Thanks so much for your assistance