Refersh header while interractive session is in progress

[nbanb]

Hi Vi

Hope you're well. Thanks again for the 'exit_on_specific_byte' function you develop for my previous use case in order to escape interractive session when the terminal is in raw mode.

Today, I have another need I didn't succeed to make work with the actual options, maybe I didn't find or understand the option.

Here is the command I'm using :

SSL_CERT_FILE=/dev/shm/fbx-cacert-ws websocat -H "X-Fbx-App-Auth: $_SESSION_TOKEN" --origin https://fbx.fbx.lan --protocol "chat, superchat" -E --binary tcp-listen:127.0.0.1:5900 wss://fbx.fbx.lan/api/v10/vm/17/vnc

# Second launching TigerVNC-viewer (providing vncviewer command) : 
# optvnc+=(...)
optvnc+=(-shared -geometry 1920x1080 -display $DISPLAY 127.0.0.1::5900 )
vncviewer "${optvnc[@]}"

The problem is that the _SESSION_TOKEN I pass by the header is only valid 30 minutes. I'm using websocat to connect VM screen from QEMU VNC over websocket and also to access VM serial console as an out-of-band access through QEMU websocket.

I have a function which refresh the _SESSION_TOKEN every 1750 seconds (30 min - 50 seconds) but how to pass the new token to websocat without relaunching the whole command ? relaunching the command would result in a disconnection from the VM screen or console and to a reconnection which is not possible when someone is using the system to manage or to work on some VM (I cannot force them to disconnect/reconnect every 30 minutes)

So my questions are : How to make websocat refresh the token without relaunching the connection ? Is there an option in websocat I haven't seen ?

Here are the functions which refresh the token in the background

ws_session () {
sleep .5
#debug
#echo $(date) >./wstemp.log
#echo $_SESSION_TOKEN >>./wstemp.log
#/debug
while [[ $(pgrep websocat) ]] ;
        do
                # Refresh token every 1750 second (expire every 1800s)
                for ((i=1;i<3500;i++));
                        do
                        pgrep websocat >/dev/null || break
                        sleep 0.5
                        ((i++))
                done
                source ${BASH_SOURCE} && relogin_freebox
                #debug
                #echo $(date +"%Y%m%d %H:%M:%S") >>./wstemp.log
                #echo  $(pgrep websocat) >>./wstemp.log
                #echo $_SESSION_TOKEN >>./wstemp.log
                #/debug
        done
exit 200 # this function is never sourced directly from a tty but from websocat_session() 
}

websocat_session () {
ws_session & 2>&1 >/dev/null
}

and after I want to launch websocat this way :

    req="${optsttys[@]}; ${optssl[@]} websocat ${options[@]} ${optws[@]} ${wsurl}"
    websocat_session \
    && bash -c "${req[@]} &" \
    && local wspid=$(pgrep websocat) \
    && vncviewer "${optvnc[@]}"

# req[@] contains : 
# SSL_CERT_FILE=/dev/shm/fbx-cacert-ws websocat -H "X-Fbx-App-Auth: $_SESSION_TOKEN" --origin https://fbx.fbx.lan --protocol "chat, superchat" -E --binary tcp-listen:127.0.0.1:5900 wss://fbx.fbx.lan/api/v10/vm/17/vnc

Is there a way for websocket to open a pipe where my websocat_session() function can write the new token in order that websocat refresh the connection in the background without breaking access to the VM screen or console ?

Thanks for help Kind regards nbanba

[vi]

The problem is that the _SESSION_TOKEN I pass by the header is only valid 30 minutes.

Does expiration of the token closes existing WebSocket connection or just prevents new WebSocket connections with the expired token?

If the former then I don't see the way to preserve forwarded VNC connection while reconnecting to WebSocket, unless VNC itself has some graceful reconnect feature.

Is there a reference (e.g. browser) implementation of that VNC client? What happens after 30 minutes of using it? Does it just keep existing connection despite of it being initiated using expired token or it reconnects under the hood?

---

How to make websocat refresh the token without relaunching the connection?

The question does not make sense, as the token (i.e. HTTP request header) is only used for initiating the connection, not keeping it open.

There can be some hacky scheme to keep incoming TCP connection open while reconnecting WebSocket connection, but it is hard to do so without losing some data when the reconnection happens.

---

What if just start client stack in a loop? Just deal with the window closing and reopening once in a while?

[nbanb]

Dear Vi

Thanks again for help and answer which helps me understanding the underlying issue :)

Your answer helps me diving in the issue and maybe I have/had a lack of understanding of the issue when opening it (or it's a bug) but the trouble is that the solution I'm forced to use have no support, a very poor documentation and different behaviors are not documented as needed...

But from my understanding, we could dissociate 2 things:

• the websocket pipe I connect after opening it with an authentified API call (the API call authentified by the $_SESSION_TOKEN open the websocket in which it's exposed the VNC screen ou the serial console device of the VM)
• The VNC or serial devices and connection inside the websocket pipe.

Following to your last answer, I did have a deeper look and notice this behavior : after 30 minutes, the connection continue to work but new connections with the expired token failed. So from my last understanding, it don't close the websocket pipe after 30 minutes, it's only discarding new connection with this token) But after more than 30 minutes if the serial terminal or the vnc session becomes idle, it's impossible to make it work again (to refresh the vnc screen or serial console) without killing websocat and re-issueing the whole API command which login, open websocket on the hypervisor, expose the VNC device or the serial console device of the vm through this websocket, and reconnecting the device from the function.

Having a look at the very poor API log/output during the issue let see the following message: "You must connect to access this function". If I reconnect the websocket, the vnc screen or the serial console started to work again immediately as they should (the VM is not in an idle state, only the connection to its vnc device or serial console was)

From your last answer, I think there could be the following issue / behavior at the hypervisor side : Maybe it keeps the websocket open until it see some traffic and if it becomes idle, it stop working (not killing the websocket, websocat is still running when it's happened), and when some traffic arrives again, it forces a reconnection which could explain the log message and the manualy reconnect action needed to access the VM again.

So from our discussion, and thanks again for your explanation of websocat behavior with headers token, it helps me understanding the underlying behavior of the hosting device websocket API (I cannot have help from its vendor, they don't provide any help nor documentation at all)

I think maybe a sort or keepalive in the websocket could be enough to keep the connection even if the serial console or the vnc screen is idle.

As I need to be agnostic of the operating system running inside the VM, I cannot generate a stealth activity to try to keepalive the connection, I need it works whatever OS or configuration's running inside the VM. Is there a way to test to have a low level keepalive in the websocket send by websocat during the whole session even if the serial connection or the vnc device screen become idle ? Maybe it would solve this issue...

Thanks and kind regards nbanba

[vi]

I think maybe a sort or keepalive in the websocket could be enough to keep the connection even if the serial console or the vnc screen is idle.

Try --ping-interval 30.

[nbanb]

Test is in progress... Thanks

[nbanb]

it's now 2h , still working ... sounds good ! I let the VM (serial + VNC) connection in a idle state for at least 2h more and I let you know

After the next test will be to set the standby on the desktop at 30 min and to wait 1h and see if the Desktop still wake up as it should.

I let you know Thanks again

[nbanb]

Hi Vi

Thanks again for help It's now 6 hours and both connections (serial & vnc) are still working

Having a look on the different 'idle' timeout on the desktop show that now t's working as expected

nba@debian:~$ gsettings describe org.gnome.settings-daemon.plugins.power sleep-inactive-ac-timeout
The amount of time in seconds the computer on AC power needs to be inactive before it goes to sleep. A value of 0 means never.
nba@debian:~$ gsettings get org.gnome.settings-daemon.plugins.power sleep-inactive-ac-timeout 
1200
nba@debian:~$ gsettings describe org.gnome.desktop.session idle-delay
The number of seconds of inactivity before the session is considered idle.
nba@debian:~$ gsettings get org.gnome.desktop.session idle-delay
uint32 300

So the option --ping-interval 30 solve the issue. Thanks

After, going back to the original subject (refreshing the header while interractive session is in progress), after thinking, it could be nice (if not already exist) that websocat provide an interface to the user where we can dynamically send options / parameters to an already runing session, for example modifying the --ping-interval value dynamically or every parameters or temporary open a pipe in read or write or both which allow to interract with the current session and close it after without closing the original session
For my project, it could help me detect when a user is login on a VM using the serial console and he launch gnome desktop from the VM shell to automatically launch the VNC client to connect graphical desktop through the vnc websocket or to provide the option to automatically launch back the serial console connection when the vnc device is removed after X shutdown...

Thanks again Kind regards nbanba

[nbanb]

It could also help handling those kind of messages :

[WARN  websocat::ws_peer] Received a pong with a strange content from websocket

Regards, nbanba

[vi]

It could also help handling those kind of messages : Received a pong with a strange content from websocket

What could help?

I expect this message to indicate non-conformant WebSocket implementation on the other side - it seems to mangle ping replies.

If everything works properly, this message should not be reachable.

[vi]

... modifying the --ping-interval value dynamically ...

I don't really understand. Why would you want to edit it? To cause eventual disconnection?

(Note: haven't read that comment in full yet).

... or every parameters ...

This may come in some future Websocat version after major redesign. Don't expect it soon.

open a pipe ... session ... and close it after ...

Save a pidfile and terminate Websocat process?

(not fully comprehended the quoted passage)

---

Can you draw a diagram of what is happening in those sessions and how is Websocat participating and so on?

Maybe the complexity has grown beyond level reasonable for hacking it together with CLI tools and you should just write a dedicated client?

[nbanb]

Hi Vi

Thanks for answer

I was thinking of something like that, letting inspect the sessions contents inside the websocket tunnel and interract with

              ws pipe
     user 1 <===========> server (exposing shared devices like VNC multi client)
        ...      ||
   (userN) <====>|| 
                 ||<====> sessions inspections (and action detection/response)
   control <====>|| 
                 ||====> log / debug (with it's own verbosity)
                ...

But you may be right, maybe I should write a dedicated client...

Does websocat3 would allow dynamic interractions on a running connection ?

Kind regards, nbanba

[vi]

Websocat3 is probably just abandoned, I sometimes experiment with (currently unpublished) Websocat4 based on rhai scripts. Even if Websocat3 was more usable, I doubt it could do something like "dynamic interractions on a running connection" (I don't understand what it really means).

---

The diagram is not clear, especially the vertical || line. Is it a hub that duplicates everything? It is not clear whether "ws pipe" is one WebSocket connection or a series of potentially parallel WebSocket connections. If just one then connecting multiple VNC clients to one WebSocket (effectively TCP) connection looks strange.

It is not clear what is a "session".

From earlier posts if follows that there may be multiple connections - one for VNC, the other for serial port or something. It is not depicted on the diagram.

You can also try building a sequence diagram that shows what happens and in what order when first client connects, when sessions gets renewed, when subsequent client connects, how it is supposed to disconnect and so on.

[nbanb]

Dear Vi Thanks again for answer and help... So nice ! OK, I will try to make a sequence diagram and go back here with.

To clarify what is 'session inspection': it would be a pipe where I can retrieve and interpret every bits transfering in both direction in the websocket pipe and where I can analyse the content (deep inspection) to be able to apply conditional actions, and to dynamically apply some kind of security profiles (detecting security threats, non standard comportement,...) By design, I would my library accept that security solutions could be plug and interract on the current flow without disturbing it, except when a threat is detected

To explain the point on the multi connection, for each running VM, the hypervisor expose 2 kinds of devices to the user (after the user asked for with an API call) :

• VM console through a serial console device
• VM screen through a shared & multiuser / multisession VNC device, allowing multiple users or one user to have multiple access to vnc screens on the same VM a bit like in a "systemd multi-seat configuration".

I'll be back with a better diagram

Kind regards nbanba

[vi]

... retrieve and interpret every bits transfering in both direction in the websocket pipe

log: overlay prints incoming and outgoing buffers to stderr, e.g.

$ websocat -b - log:ws://ws.vi-server.org/mirror
asdf
WRITE 5 "asdf\n"
READ 5 "asdf\n"
asdf
1234
WRITE 5 "1234\n"
READ 5 "1234\n"
1234

This is mostly intended for debugging.

... deep inspection ... security profiles ... security threats ...

This suggests using a dedicated tcp-to-websocket proxy, not combining flexible components into a solution.

Various overlays and other tricky options in Websocat are primarily intended for development, debugging and one-off quick experiments. To use Websocat as a permanent installation handling untrusted data one would require to inspect parts of Websocat used for the task. Implementing a separate, more specific tool may be easier than such inspection.

[nbanb]

Hi Vi

Thanks again for answer and help. I will try the log option.

I think you're right, for handling untrusted data and making inspections, maybe a dedicated tool would be easier and more safe by design.

Kind regards nbanba

[nbanb]

Dear Vi

Thanks again for answer & help

As https://www.rfc-editor.org/rfc/rfc6455 describes a not too difficult protocole (maybe section 5.2 of RFC6455 which explane data masking is the hardest part), it was decided to develop a specific tool to be able to inspect the packet flow inside the websocket pipe and to be able to interract with from a third party engine.

This case was solved by the --ping-timeout option in websocat, so I think it could be closed.

Kind regards nbanba