search

golioth / grafana-websocket-plugin

Apache License 2.0
32 stars 13 forks source link

websocket issues with basic auth within HTTPS #21

Open bijwaard opened 2 years ago

bijwaard commented 2 years ago

Dear Golioth,

Thanks again for your plugin. I have been struggling to get basic authentication and HTTPS to work, via an apache proxy. I got this working with my HTTPS test client. However, the connection is not set-up when I try the same with the websocket-plugin under grafana, could it be disgarded within the websocket plugin?

The websocket-plugin works fine when grafana and the websocket are used directly (not web-proxied), but this would not be secure enough to open up to the wolves on the Internet.

The reason to use basic authentication (Oauth2 may be acceptable alternative), is that users with access to grafana should only be allowed to get the live data for a selection of sensors, the X-auth key would allow usage to every user that can select/use the datasource in a dashboard. Furthermore, changing access for users becomes easier with basic authentication since that can be administrated centrally in the proxy server that handles the authentication members&groups, else the X-auth key may need to change for each member/group change, and probably needs to be different for each datasource, i.e. smells like a maintenance nightmare. Additionally, some users want to use the same websocket for other purposes without Grafana, so it still needs to be protected independent from Grafana.

Regular websocket test client connection using test html client works though a primary (HTTPS+auth) and secondary web proxy (ssh-tunneled HTTP site with sensors):

Primary web-proxy with HTTPS and authentication to wss://<domain>:<port>:

192.168.2.254 - dennis [13/Sep/2022:09:11:08 +0000] "GET /nl/enschede/lab/busbar/testws/rms_min HTTP/2.0" 200 4998 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
192.168.2.254 - dennis [13/Sep/2022:09:11:08 +0000] "GET /nl/enschede/lab/busbar/testws/rms_min HTTP/2.0" 200 4998 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"

Secondary web-proxy with HTTP proxying for a remote sensor group:

::1 - - [13/Sep/2022:09:12:09 +0000] "GET /busbar/testws/rms_min?testws HTTP/1.1" 200 5077 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"

[Tue Sep 13 09:12:09.438837 2022] [proxy:debug] [pid 347017:tid 281470950175104] proxy_util.c(2532): AH00942: http: has acquired connection for (localhost)
[Tue Sep 13 09:12:09.441325 2022] [proxy:debug] [pid 347017:tid 281470950175104] proxy_util.c(3277): AH02824: http: connection established with [::1]:5000 (localhost)
[Tue Sep 13 09:12:09.441995 2022] [proxy:debug] [pid 347017:tid 281470950175104] proxy_util.c(3463): AH00962: http: connection complete to [::1]:5000 (localhost)
[Tue Sep 13 09:12:09.455120 2022] [proxy:debug] [pid 347017:tid 281470950175104] proxy_util.c(2547): AH00943: http: has released connection for (localhost)
[Tue Sep 13 09:12:09.455396 2022] [proxy:debug] [pid 347017:tid 281470950175104] proxy_util.c(3387): [remote ::1:5000] AH02642: proxy: connection shutdown
[Tue Sep 13 09:12:09.536440 2022] [proxy:debug] [pid 347017:tid 281471302472064] proxy_util.c(2547): AH00943: http: has released connection for (localhost)
[Tue Sep 13 09:12:09.536694 2022] [xml2enc:debug] [pid 347017:tid 281471302472064] mod_xml2enc.c(195): [client ::1:41466] AH01430: Content-Type is text/plain; charset=utf-8
[Tue Sep 13 09:12:09.536819 2022] [xml2enc:info] [pid 347017:tid 281471302472064] [client ::1:41466] AH01431: Got charset utf-8 from HTTP headers
[Tue Sep 13 09:12:09.543584 2022] [authz_core:debug] [pid 347088:tid 281471445082496] mod_authz_core.c(843): [client ::1:41470] AH01628: authorization result: granted (no directives)
[Tue Sep 13 09:12:09.543873 2022] [core:debug] [pid 347088:tid 281471445082496] protocol.c(2455): [client ::1:41470] AH03155: select protocol from h2,h2c,http/1.1, choices=WebSocket for server ::1
[Tue Sep 13 09:12:09.544014 2022] [http2:debug] [pid 347088:tid 281471445082496] h2_switch.c(90): [client ::1:41470] AH03085: upgrade without HTTP2-Settings declined
[Tue Sep 13 09:12:09.544373 2022] [core:debug] [pid 347088:tid 281471445082496] protocol.c(2519): [client ::1:41470] AH03157: selected protocol=(none)
[Tue Sep 13 09:12:09.544535 2022] [proxy:debug] [pid 347088:tid 281471445082496] mod_proxy.c(1503): [client ::1:41470] AH01143: Running scheme ws handler (attempt 0)
[Tue Sep 13 09:12:09.544667 2022] [proxy:debug] [pid 347088:tid 281471445082496] proxy_util.c(2532): AH00942: ws: has acquired connection for (busbar)
[Tue Sep 13 09:12:09.544946 2022] [proxy:debug] [pid 347088:tid 281471445082496] proxy_util.c(2588): [client ::1:41470] AH00944: connecting ws://busbar:5000/ws/rms_min to busbar:5000
[Tue Sep 13 09:12:09.546044 2022] [proxy:debug] [pid 347088:tid 281471445082496] proxy_util.c(2811): [client ::1:41470] AH00947: connected /ws/rms_min to busbar:5000
[Tue Sep 13 09:12:09.548749 2022] [proxy:debug] [pid 347088:tid 281471445082496] proxy_util.c(3277): AH02824: ws: connection established with 192.168.11.2:5000 (busbar)
[Tue Sep 13 09:12:09.548981 2022] [proxy:debug] [pid 347088:tid 281471445082496] proxy_util.c(3463): AH00962: ws: connection complete to 192.168.11.2:5000 (busbar)
[Tue Sep 13 09:12:09.553693 2022] [proxy_http:debug] [pid 347088:tid 281471445082496] mod_proxy_http.c(1478): [client ::1:41470] AH10239: HTTP: tunneling protocol websocket
[Tue Sep 13 09:12:09.591900 2022] [authz_core:debug] [pid 347017:tid 281470933389696] mod_authz_core.c(843): [client ::1:41474] AH01628: authorization result: granted (no directives)
[Tue Sep 13 09:12:09.592115 2022] [proxy:debug] [pid 347017:tid 281470933389696] mod_proxy.c(1503): [client ::1:41474] AH01143: Running scheme http handler (attempt 0)
[Tue Sep 13 09:12:09.592186 2022] [proxy:debug] [pid 347017:tid 281470933389696] proxy_util.c(2532): AH00942: http: has acquired connection for (localhost)

Connection accepted in my websocket server (rtd_web):

rtd_web: 2022/09/13 09:12:09 &{GET /ws/rms_min HTTP/1.1 1 1 map[Accept:[*/*] Accept-Language:[en-US,en;q=0.5] Authorization:[Basic Z***] Cache-Control:[no-cache] Connection:[Upgrade] Dnt:[1] Origin:[https://<domain>:<port>] Pragma:[no-cache] Sec-Fetch-Dest:[websocket] Sec-Fetch-Mode:[websocket] Sec-Fetch-Site:[same-origin] Sec-Websocket-Extensions:[permessage-deflate] Sec-Websocket-Key:[X**] Sec-Websocket-Version:[13] Upgrade:[WebSocket] User-Agent:[Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0] X-Forwarded-For:[192.168.2.254, ::1] X-Forwarded-Host:[<domain>:<port>, <domain>:<port>] X-Forwarded-Server:[<domain>, ::1]] {} <nil> 0 [] false <domain>:<port> map[] map[] <nil> map[] 192.168.11.1:35042 /ws/rms_min <nil> <nil> <nil> 0x400032e570}
rtd_web: 2022/09/13 09:12:09 ################Request Headers################
Authorization : [Basic Z***]
Sec-Websocket-Extensions : [permessage-deflate]
Pragma : [no-cache]
X-Forwarded-For : [192.168.2.254, ::1]
X-Forwarded-Server : [<domain>, ::1]
Accept : [*/*]
Sec-Websocket-Version : [13]
Dnt : [1]
X-Forwarded-Host : [<domain>:<port>, <domain>:<port>]
Upgrade : [WebSocket]
User-Agent : [Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0]
Origin : [https://<domain>:<port>]
Sec-Websocket-Key : [X***]
Sec-Fetch-Dest : [websocket]
Sec-Fetch-Mode : [websocket]
Sec-Fetch-Site : [same-origin]
Cache-Control : [no-cache]
Connection : [Upgrade]
Accept-Language : [en-US,en;q=0.5]
rtd_web: 2022/09/13 09:12:09 Received new Client connection
rtd_web: 2022/09/13 09:12:09 Client upgrade to WS success
rtd_web: 2022/09/13 09:12:09 Subscribing to VT_DSPs_Median_rms:rms_min on channel 0x40001000c0
rtd_web: 2022/09/13 09:12:09 The Received Message type is 1 
rtd_web: 2022/09/13 09:12:09 The Received Message is [72 105 32 83 101 114 118 101 114 46 32 73 109 32 74 83 32 67 108 105 101 110 116]

Unfortunately, grafana websocket connection breaks and does not end up in any connection to my websocket server:

192.168.2.254 - dennis [13/Sep/2022:10:54:34 +0000] "POST /nl/enschede/lab/nl/enschede/lab/gf/api/ds/query HTTP/2.0" 200 387 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
192.168.2.254 - dennis [13/Sep/2022:10:54:34 +0000] "POST /nl/enschede/lab/nl/enschede/lab/gf/api/ds/query HTTP/2.0" 200 394 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
192.168.2.254 - dennis [13/Sep/2022:10:54:34 +0000] "GET /nl/enschede/lab/nl/enschede/lab/gf/api/annotations?from=1663066174558&to=1663066474558&limit=100&matchAny=false&dashboardUID=IK5EXKM4z HTTP/2.0" 200 223 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
192.168.2.254 - dennis [13/Sep/2022:10:54:42 +0000] "GET /nl/enschede/lab/gf/api/live/ws HTTP/1.1" 400 5613 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"

And on secondary proxy:

::1 - - [13/Sep/2022:10:54:34 +0000] "POST /nl/enschede/lab/gf/api/ds/query HTTP/1.1" 200 487 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
::1 - - [13/Sep/2022:10:54:34 +0000] "POST /nl/enschede/lab/gf/api/ds/query HTTP/1.1" 200 494 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
::1 - - [13/Sep/2022:10:54:34 +0000] "GET /nl/enschede/lab/gf/api/annotations?from=1663066174558&to=1663066474558&limit=100&matchAny=false&dashboardUID=IK5EXKM4z HTTP/1.1" 200 319 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
::1 - - [13/Sep/2022:10:54:42 +0000] "GET /nl/enschede/lab/gf/api/live/ws HTTP/1.1" 400 338 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"

[Tue Sep 13 10:54:33.827997 2022] [proxy:debug] [pid 349793:tid 281472510431616] proxy_util.c(2547): AH00943: http: has released connection for (localhost)
[Tue Sep 13 10:54:33.828138 2022] [proxy:debug] [pid 349793:tid 281472510431616] proxy_util.c(3387): [remote ::1:5000] AH02642: proxy: connection shutdown
[Tue Sep 13 10:54:34.629584 2022] [authz_core:debug] [pid 349793:tid 281472015524224] mod_authz_core.c(843): [client ::1:54676] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.629801 2022] [authz_core:debug] [pid 349863:tid 281471881306496] mod_authz_core.c(843): [client ::1:54678] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.629856 2022] [proxy:debug] [pid 349793:tid 281472015524224] mod_proxy.c(1503): [client ::1:54676] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.629955 2022] [proxy:debug] [pid 349793:tid 281472015524224] proxy_util.c(2532): AH00942: http: has acquired connection for (localhost)
[Tue Sep 13 10:54:34.630017 2022] [proxy:debug] [pid 349793:tid 281472015524224] proxy_util.c(2588): [client ::1:54676] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/ds/query to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.630015 2022] [proxy:debug] [pid 349863:tid 281471881306496] mod_proxy.c(1503): [client ::1:54678] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.630084 2022] [proxy:debug] [pid 349793:tid 281472015524224] proxy_util.c(2811): [client ::1:54676] AH00947: connected /nl/enschede/lab/gf/api/ds/query to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.630220 2022] [proxy:debug] [pid 349863:tid 281471881306496] proxy_util.c(2532): AH00942: http: has acquired connection for (localhost)
[Tue Sep 13 10:54:34.630279 2022] [proxy:debug] [pid 349863:tid 281471881306496] proxy_util.c(2588): [client ::1:54678] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/ds/query to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.630344 2022] [proxy:debug] [pid 349863:tid 281471881306496] proxy_util.c(2811): [client ::1:54678] AH00947: connected /nl/enschede/lab/gf/api/ds/query to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.675069 2022] [authz_core:debug] [pid 349863:tid 281471872913792] mod_authz_core.c(843): [client ::1:54680] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.675365 2022] [proxy:debug] [pid 349863:tid 281471872913792] mod_proxy.c(1503): [client ::1:54680] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.675466 2022] [proxy:debug] [pid 349863:tid 281471872913792] proxy_util.c(2532): AH00942: http: has acquired connection for (localhost)
[Tue Sep 13 10:54:34.675529 2022] [proxy:debug] [pid 349863:tid 281471872913792] proxy_util.c(2588): [client ::1:54680] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/annotations?from=1663066174558&to=1663066474558&limit=100&matchAny=false&dashboardUID=IK5EXKM4z to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.675604 2022] [proxy:debug] [pid 349863:tid 281471872913792] proxy_util.c(2811): [client ::1:54680] AH00947: connected /nl/enschede/lab/gf/api/annotations?from=1663066174558&to=1663066474558&limit=100&matchAny=false&dashboardUID=IK5EXKM4z to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=rms&var-signal=rms_median&var-sensor=VT&var-source=wsVT-lab%20https&from=now-5m&to=now
[Tue Sep 13 10:54:34.676565 2022] [proxy:debug] [pid 349863:tid 281471872913792] proxy_util.c(3277): AH02824: http: connection established with [::1]:3000 (localhost)
[Tue Sep 13 10:54:34.676800 2022] [proxy:debug] [pid 349863:tid 281471872913792] proxy_util.c(3463): AH00962: http: connection complete to [::1]:3000 (localhost)
[Tue Sep 13 10:54:34.737240 2022] [proxy:debug] [pid 349793:tid 281472015524224] proxy_util.c(2547): AH00943: http: has released connection for (localhost)
[Tue Sep 13 10:54:34.738698 2022] [proxy:debug] [pid 349863:tid 281471881306496] proxy_util.c(2547): AH00943: http: has released connection for (localhost)
[Tue Sep 13 10:54:34.749325 2022] [proxy:debug] [pid 349863:tid 281471872913792] proxy_util.c(2547): AH00943: http: has released connection for (localhost)

Kind regards, Dennis

bijwaard commented 2 years ago

I noticed a lot of connections to the default live datasource in the grafana and apache log, could grafana be confused and try to use the default live datasource instead of my configured websocket datasource? Both the websocket and grafana are proxied on the same HTTPS URL <domain>:<port>, however the datasource wsVT-lab%20https has wss://<domain>:<port>/nl/enschede/lab/busbar/ws/ configured in the host field, while the live datasource has path /nl/enschede/lab/gf/api/live/ws on this URL, since the grafana root_url contains the path /nl/enschede/lab/gf.

[Tue Sep 13 14:31:17.372481 2022] [authz_core:debug] [pid 357717:tid 281471872913792] mod_authz_core.c(843): [client ::1:47382] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.372815 2022] [authz_core:debug] [pid 357717:tid 281473231876480] mod_authz_core.c(843): [client ::1:47384] AH01628: authorization result: granted (no directives), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.372875 2022] [proxy:debug] [pid 357717:tid 281471872913792] mod_proxy.c(1503): [client ::1:47382] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373053 2022] [proxy:debug] [pid 357717:tid 281473231876480] mod_proxy.c(1503): [client ::1:47384] AH01143: Running scheme http handler (attempt 0), referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373215 2022] [proxy:debug] [pid 357717:tid 281471872913792] proxy_util.c(2596): [client ::1:47382] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/annotations?from=1663079417335&to=1663079477336&limit=100&matchAny=false&dashboardUID=IK5EXKM4z to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373313 2022] [proxy:debug] [pid 357717:tid 281471872913792] proxy_util.c(2819): [client ::1:47382] AH00947: connected /nl/enschede/lab/gf/api/annotations?from=1663079417335&to=1ny=false&dashboardUID=IK5EXKM4z to localhost:3000, referer: https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20
[Tue Sep 13 14:31:17.373353 2022] [proxy:debug] [pid 357717:tid 281473231876480] proxy_util.c(2596): [client ::1:47384] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/ds/query to l://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
[Tue Sep 13 14:31:17.373643 2022] [proxy:debug] [pid 357717:tid 281473231876480] proxy_util.c(2819): [client ::1:47384] AH00947: connected /nl/enschede/lab/gf/api/ds/query to localhost:3000, refer <domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now
::1 - - [13/Sep/2022:14:31:17 +0000] "GET /nl/enschede/lab/gf/api/annotations?from=1663079417335&to=1663079477336&limit=100&matchAny=false&dashboardUID=IK5EXKM4z HTTP/1.1" 200 319 "<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&var-sensor=VT&var-source=wsVT-lab%20https&from=now-1m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
::1 - - [13/Sep/2022:14:31:17 +0000] "POST /nl/enschede/lab/gf/api/ds/query HTTP/1.1" 200 486 "https://<domain>:<port>/nl/enschede/lab/gf/d/IK5EXKM4z/test_ws?orgId=1&var-signal=RS&vlab%20https&from=now-1m&to=now" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
[Tue Sep 13 14:31:50.718382 2022] [proxy:debug] [pid 357717:tid 281473181520256] proxy_util.c(2596): [client ::1:47390] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/live/ws to lo
[Tue Sep 13 14:31:50.718458 2022] [proxy:debug] [pid 357717:tid 281473181520256] proxy_util.c(2819): [client ::1:47390] AH00947: connected /nl/enschede/lab/gf/api/live/ws to localhost:3000
::1 - - [13/Sep/2022:14:31:50 +0000] "GET /nl/enschede/lab/gf/api/live/ws HTTP/1.1" 400 338 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0"
[Tue Sep 13 14:32:50.903748 2022] [proxy:debug] [pid 357715:tid 281472678220160] proxy_util.c(2596): [client ::1:47404] AH00944: connecting http://localhost:3000/nl/enschede/lab/gf/api/live/ws to lo
[Tue Sep 13 14:32:50.903965 2022] [proxy:debug] [pid 357715:tid 281472678220160] proxy_util.c(2819): [client ::1:47404] AH00947: connected /nl/enschede/lab/gf/api/live/ws to localhost:3000

Kind regards, Dennis

bijwaard commented 1 year ago

Hi Again,

Apparently, one of my problems was wrongly configured proxying of the grafana api/live/wx path in the apache proxy. This could be remedied with the following rule(s) in the secondary apache proxy (and similarly in the primary Internet-facing proxy):

        Define GrafanaPath /nl/enschede/lab/gf
        ProxyPassMatch    "${GrafanaPath}/api/live/(.*)$"      ws://localhost:3000${GrafanaPath}/api/live/$1
        ProxyPass          ${GrafanaPath}      http://localhost:3000${GrafanaPath}
        ProxyPassReverse   ${GrafanaPath}      http://localhost:3000${GrafanaPath}

After this fix, the live streaming was working with websockets local to grafana, such as ws://busbar/ws, but not with basic authentication. I was initially under the impression that the configured websocket was handled browser-side, but apparently it is handled server-side (using api/live/ws websocket within grafana localhost:3000 that re-streams all configured websockets to the browser). It is currently not possible to give basic-auth credentials with the websocket datasource configuration, so this won't work (yet) with server-side websockets.

It would be nice to also have browser-side support for websockets, such that the authentication would be handled by the browser and the authentication doesn't need to be configured&maintained in grafana.

Using brower-side websockets would also reduce the CPU load on the grafana server, e.g. on my embedded PC (nanoPI neo3), server-side websockets take +/- 10% of CPU core per 50Hz stream. With mulitple clients selecting different streams this may not scale. For example with four 50Hz streams:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                     
 472512 grafana   20   0  718956  20212  10816 S  24.4   2.0   1104:43 /var/lib/grafana/plugins/golioth-websocket-datasource/gpx_websocket_linux_arm64                                             
 415835 grafana   20   0 2057884  98876  33204 S  19.8   9.9   2292:06 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:de+

Kind regards, Dennis

chireia commented 1 year ago

Hey @bijwaard , sorry about the lack of responses! Great work, seems like you got it working, awesome.

When we first implemented the plugin, the only way to do authentication was by using the server-side handled by Grafana. Being transparent, I'm not aware that any changes on that behavior. Do you have any suggestions and/or experiences on doing that using only the client-side? Since Grafana introduced "Grafana Live" on their v8.0, plugin-wise development guides have being pretty basic and rudimentary. So we'd love to hear from you.

bijwaard commented 1 year ago

Hi @chireia,

Thanks for your message. I didn't pursue this much further, grafana seems to phase out browser-side support for other datasources as well (influxdb, prometheus, etc.). In my view browser-side support gives much more flexibility than forcing all data through the grafana server, and sometimes the grafana server may not be able to reach the datasource itself.

It is a pity that grafana (server-side) live streaming takes so much resources, and that it sometimes takes more than 10 seconds to start streaming when the dashboard is opened, and it has hick-ups when the dashboard refreshes other datasources. Our alternative is dashboard refresh at e.g. 1s interval, with our own JSON plugin back-end written in golang.

P.s. browser-side websockets with the same webserver-proxy works fine outside grafana, also with basic authentication.

Kind regards, Dennis