search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
13.95k stars 3.53k forks source link

Proxy Websocket authentication using websocket in browser #5598

Open waxzce opened 4 years ago

waxzce commented 4 years ago

Websocket API in a browser does not allow to manage header https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/WebSocket

The authentication using plain HTTP is not possible on this case, and it seems that is what the pulsar implement at the moment https://github.com/apache/pulsar/blob/master/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/AbstractWebSocketHandler.java#L70

There are mainly 4 solutions used out there to authenticate WS:

  1. All in the query string (do not want to argue here, seems to be a bad bad idea)
  2. Send a first message after connection containing the auth and close the WebSocket if there is no auth message for X seconds (I'm also not very fan of this)
  3. In the URL, you can add wss://user:password@pulsarhost/ and it will be treated as HTTP basic auth, a wss://my_token@pulsarhost/ will be seen as the user ill be the token and the password empty. Can work despite the ugly semantic.
  4. At the opening of browser WebSocket, we can pass on arguments a string array which will be sent as Sec-WebSocket-Protocol header and is available downstream to the authImpl, so each auth plugin can manage it. (there is some limitation so using base64 will be mandatory, but easy wrapping)

So, how do you want to manage this? Manage it explicitly on the WebSocket proxy or let people manage?

My personal opinion is option 4, but if we do that, best will be to make it work on default auth plugin, and add it to the documentation.

vicaya commented 4 years ago

Ideally, the browser websocket api needs to be enhanced to support additional headers per websocket RFC 6455 (section 4.1). OTOH, IMO, 3 (using basic auth info as token) is the most practical and requires minimal change at both client and server side. Using subprotocols for auth is even more hackish and requires more substantial changes to existing clients and servers.

Geal commented 4 years ago

FYI I just implemented it in our authentication provider, it's straightforward to add https://github.com/CleverCloud/biscuit-pulsar/blob/master/src/main/java/com/clevercloud/biscuitpulsar/BiscuitAuthenticationPlugin.java#L68-L73

roackb2 commented 3 years ago

I'm successful to authorize a Node.js WebSocket client with JWT authentication and authorization enabled on Pulsar, but I have issue establishing the connection.

By using npm ws module, I was able to set Authorization Bearer like this:

this.wsUrl = `wss://${opts.host}:${opts.port}/ws/v2/consumer/persistent/${opts.tenant}/${opts.namespace}/${opts.topic}/${opts.subscription}`;
this.wsOpts = {
    headers: {
        Authorization: `Bearer ${opts.token}`
    }
}
this.websocket = new WebSocket(this.wsUrl, this.wsOpts);

The resulting url is like wss://localhost:8443/ws/v2/consumer/persistent/my-tenant/my-namespace/my-topic/sub-1

And viewing the log of Pulsar, I could see that authorization is successful:

08:59:55.403 [pulsar-web-69-3] INFO  org.apache.pulsar.websocket.AbstractWebSocketHandler - [172.17.0.1:53360] Authenticated WebSocket client topology-admin on topic persistent://my-tenant/my-namespace/my-topic

However, the following error shows up:

08:59:55.617 [pulsar-client-io-94-6] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x618b3eb3, L:/127.0.0.1:47428 - R:localhost/127.0.0.1:6650]] Connected to server
08:59:55.633 [pulsar-io-51-1] INFO  org.apache.pulsar.broker.service.ServerCnx - New connection from /127.0.0.1:47428
08:59:55.716 [pulsar-io-51-1] WARN  org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:47428] Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 369295620 - discarded
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:503)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:489)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:376)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:419)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:332)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

08:59:55.752 [pulsar-io-51-1] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /127.0.0.1:47428
08:59:55.750 [pulsar-client-io-94-6] WARN  org.apache.pulsar.client.impl.ClientCnx - Error during handshake
java.nio.channels.ClosedChannelException: null
    at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1076) [io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
08:59:55.822 [pulsar-client-io-94-6] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0x618b3eb3, L:/127.0.0.1:47428 ! R:localhost/127.0.0.1:6650] Disconnected
08:59:55.827 [pulsar-client-io-94-6] WARN  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x618b3eb3, L:/127.0.0.1:47428 ! R:localhost/127.0.0.1:6650]] Connection handshake failed: org.apache.pulsar.client.api.PulsarClientException: Connection already closed
08:59:55.933 [pulsar-external-listener-95-1] WARN  org.apache.pulsar.client.impl.PulsarClientImpl - [topic: persistent://my-tenant/my-namespace/my-topic] Could not get connection while getPartitionedTopicMetadata -- Will try again in 100 ms
08:59:55.955 [pulsar-client-io-94-8] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x13fe0546, L:/127.0.0.1:47430 - R:localhost/127.0.0.1:6650]] Connected to server
08:59:55.960 [pulsar-io-51-2] INFO  org.apache.pulsar.broker.service.ServerCnx - New connection from /127.0.0.1:47430
08:59:55.972 [pulsar-io-51-2] WARN  org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:47430] Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 369295620 - discarded
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:503)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:489)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:376)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:419)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:332)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

08:59:55.994 [pulsar-io-51-2] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /127.0.0.1:47430
08:59:56.005 [pulsar-client-io-94-8] WARN  org.apache.pulsar.client.impl.ClientCnx - Error during handshake
java.nio.channels.ClosedChannelException: null
    at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1076) [io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
08:59:56.021 [pulsar-client-io-94-8] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0x13fe0546, L:/127.0.0.1:47430 ! R:localhost/127.0.0.1:6650] Disconnected
08:59:56.024 [pulsar-client-io-94-8] WARN  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x13fe0546, L:/127.0.0.1:47430 ! R:localhost/127.0.0.1:6650]] Connection handshake failed: org.apache.pulsar.client.api.PulsarClientException: Connection already closed
08:59:56.218 [pulsar-external-listener-95-1] WARN  org.apache.pulsar.client.impl.PulsarClientImpl - [topic: persistent://my-tenant/my-namespace/my-topic] Could not get connection while getPartitionedTopicMetadata -- Will try again in 191 ms

With some study, I know that TooLongFrameException is most likely caused by mismatch of whether using TLS between the client and the broker, and by viewing the log, I see this part: 

08:59:55.617 [pulsar-client-io-94-6] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x618b3eb3, L:/127.0.0.1:47428 - R:localhost/127.0.0.1:6650]] Connected to server

It seems that ConnectionPool is connecting secure WebSocket to non-TLS boker service port 6650, can anyone give me some direction how to make secure WebSocket work?

roackb2 commented 3 years ago

I solved my own issue. Since I'm running in standalone mode, but the standalone.conf template lacks of some options. I have to manually add following configurations in the conf file:

brokerClientTlsEnabled=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file://pat/to/admin-token
brokerClientTrustCertsFilePath=path/to/ca.cert.pem

This would make WebSocket successfully establish secure connection with authorization enabled.