use socket activated service to launch c8y-remote-access-plugin when using systemd

[reubenmiller]

Is your feature improvement request related to a problem? Please describe.

Using the Cumulocity IoT Remote Access feature to connect to a device via SSH is limiting as the c8y-remote-access-plugin process is spawned by the tedge-mapper-c8y process, meaning that any command used the result in restarting the tedge-mapper-c8y service will disconnect the ssh session.

Due to this limitation, the following actions are not possible:

• Updating tedge-mapper package
• Restarting tedge-mapper-c8y service

Describe the solution you'd like

On devices which use SystemD, use a socket-activated service to launch the c8y-remote-access-plugin process which is independent from the tedge-mapper-c8y service.

In order to get full independence, the following changes would have to be done:

• Change installation script to install the socket handling logic only on systems where SystemD is installed
• Use a direct connection to the Cumulocity IoT URL instead of using the local c8y proxy service (as the service is maintained by the tedge-mapper-c8y service) - (technically this will be reverting the change where we switched to using the local c8y proxy)
• Optional: Improve the c8y-remote-access-plugin handshake so that the when the process is launched via a socket, that the client publishing the message can easily verify if the c8y-remote-access-plugin was able to successfully connect to c8y or not
• Support sending a message to either a TCP or UNIX socket via the cli (possibly in the c8y-remote-access-plugin, tedge, or tedge-mapper so other potential operation handlers can take advantage of it?)

Important Notes about socket activated services

• SystemD is responsible for socket creation and listening to incoming client connections (which means very minimal overhead)
• The c8y-remote-access-plugin does not have to be running prior to any incoming client connections. SystemD will launch the process when a client tries to connect/send a message to the socket

Describe alternatives you've considered

Alternatively, the SystemD KillMode of the tedge-mapper-c8y service could be changed to process (from the default control-group), however this is a best practice as it could result in orphaned processes, and it would mean that the tedge-mapper-service would have to managed any child process launched by itself so it could prevent said orphaned processes (this is currently managed by SystemD).

Additional context

A detailed guide to how a socket activated service would work to launch an independent c8y-remote-access-plugin process:

The tedge-mapper-c8y will read the c8y_RemoteAccessConnect file, to determine which script to execute upon receiving a c8y SmartREST static message id 530, in this case it will call the publish_to_socket.sh script. The publish_to_socket.sh script will publish the received message (read from arguments) and publish the same message to a socket. It then waits for a given response from the socket to confirm that the message was received and successful.

The c8y-remote-access-plugin.socket is the systemd socket definition file which tells systemd to create a specific socket (either TCP or UNIX socket) and then when a client connects to the socket, systemd will accept the connection and spawn a new service based on the service template definition (c8y-remote-access-plugin@.service), which in turn will start the c8y-remote-access-plugin process. This results in one service per connection, however once the c8y-remote-access-plugin process exists, systemd will eventually cleanup the related service.

Below shows some files which can be used to show the moving pieces:

file: /etc/tedge/operations/c8y/c8y_RemoteAccessConnect

The c8y operation definition file which will communicate with the socket to send a new request to open up a c8y-remote-access session.

[exec]
# Custom script to publish and monitor the socket connection
command = "/usr/bin/publish_to_socket.sh --socket /run/c8y-remote-access-plugin.sock --response-ok CONNECTED --response-fail STOPPING"
topic = "c8y/s/ds"
on_message = "530"

file: /usr/bin/publish_to_socket.sh

A generic script used to communicate with a socket to publish a message and then read from the socket to confirm that the message results in a "successful action" (it just uses simple string matching to determine this, but we could do a more formal handshake as defined by c8y-remote-access-plugin).

#!/bin/sh
#
# Script to publish a message to a TCP or AF UNIX socket using socat.
# After writing to the socket, the socket will be read from to check
# if the response contains a specific string to mark the request as
# being successful or not.
#
set -e

MESSAGE=
SOCKET=
RESPONSE_OK="CONNECTED"
RESPONSE_FAIL="STOPPING"

help() {
    cat << EOT >&2
Send a message to a TCP or AF UNIX socket and read the response to determine if it
was successful or not.

USAGE

   $0 [FLAGS] <MESSAGE>

POSITIONAL ARGS

    MESSAGE                 Message to be sent to the socket

FLAGS

  --socket <address>        TCP or Unix socket path. e.g. /run/example.sock or 127.0.0.1:4444
  --response-ok <string>    String to match against to determine that the request sent to the socket
                            was received by the component reacting to the request.
  --response-fail <string>  String to match against to determine an error. This will override any
                            existing ok response.
  --help                    Show this help

EXAMPLES

   $0 --socket /run/example.socket --response-ok CONNECTED --response-fail STOPPING "530,TST_throw_crabby_exception,127.0.0.1,22,18f7c014-8180-40e0-b272-03c9dec8f327"
   # Publish a c8y-remote-access-plugin message to a socket, and check for a successful connection

EOT
}

# Parse cli options
while [ $# -gt 0 ]; do
    case "$1" in
        --socket)
            SOCKET="$2"
            shift
            ;;
        --response-ok)
            RESPONSE_OK="$2"
            shift
            ;;
        --response-fail)
            RESPONSE_FAIL="$2"
            shift
            ;;
        --help|-h)
            help
            exit
            ;;
        --*|-*)
            ;;
        *)
            MESSAGE="$1"
            ;;
    esac
    shift
done

if [ -z "$MESSAGE" ]; then
    echo "Message is empty. You MUST provide a non-empty message" >&2
    usage
    exit 1
fi

SOCAT_PREFIX="TCP:"
if [ -e "$SOCKET" ]; then
    SOCAT_PREFIX="UNIX-CONNECT:"
fi

echo "Writing message ($MESSAGE) to socket ($SOCAT_PREFIX$SOCKET)" >&2

# Write to the socket and read the response until an expected response text is found
RESPONSE=$(
    echo "$MESSAGE" | socat - "$SOCAT_PREFIX$SOCKET" | while read -r line; do
        echo "socket recv: $line" >&2
        case "$line" in
            "$RESPONSE_OK")
                echo "Detected successful response" >&2
                echo "0"
                ;;
            "$RESPONSE_FAIL")
                echo "Detected error" >&2
                echo "1"
                ;;
        esac
    done
)

# Check if request was successful
RESPONSE=$(echo "$RESPONSE" | tr -d '\n')
if [ "$RESPONSE" = "0" ]; then
    echo "Found OK response" >&2
    exit 0
fi

# Assume the request was unsuccessful
echo "Did not receive expected response from socket. expected='$WAIT_FOR_OK_RESPONSE'" >&2
exit 1

file: /lib/systemd/system/c8y-remote-access-plugin.socket

SystemD socket definition which tells SystemD to create a socket and upon connection, create a new service based on the template and pass the current socket to it:

[Unit]
Description=c8y-remote-access-plugin Socket
PartOf=c8y-remote-access-plugin.service

[Socket]
ListenStream=/run/c8y-remote-access-plugin.sock
SocketMode=0660
SocketUser=tedge
SocketGroup=tedge
Accept=yes

[Install]
WantedBy=sockets.target

file: /lib/systemd/system/c8y-remote-access-plugin@.service

SystemD service which calls the c8y-remote-access-plugin process with the information received from the socket. c8y-remote-access-plugin can send information back to the socket by just writing to stdout (Standard Output).

Note: The c8y-remote-access-plugin example has been modified so that the --child flag will read its values from stdin as denoted by the posix convention -. The StandardInput=socket systemd setting specifies that the socket should be mapped to standard input, where it can be read like any normal standard input (meaning that the process does not need to know it is reading from a socket, just stdin).

[Unit]
Description=c8y-remote-access-plugin Service
After=network.target c8y-remote-access-plugin.socket
Requires=c8y-remote-access-plugin.socket
CollectMode=inactive-or-failed

[Service]
ExecStart=/usr/bin/c8y-remote-access-plugin --child -
StandardInput=socket

[Install]
WantedBy=default.target

[reubenmiller]

I've created a branch on my fork which includes all of the files in the "Additional context" section:

• https://github.com/reubenmiller/thin-edge.io/tree/feat-remote-access-via-socket

The following changes are implemented:

• Modified c8y-remote-access-plugin to read from stdin when using --child -
• Rough test case (though the test case still needs to be modified to allow running a c8ylp command to connect to the container using the Cumulocity IoT PassThrough connection type of the Cloud Remote Access feature).

[reubenmiller]

Note: some other service managers like s6 also support socket activation, check out this PR for an example: https://github.com/eclipse/mosquitto/pull/2346/files

[rina23q]

The PR #3006 resolved the following part.

Use a direct connection to the Cumulocity IoT URL instead of using the local c8y proxy service (as the service is maintained by the tedge-mapper-c8y service) - (technically this will be reverting the change where we switched to using the local c8y proxy)

The rest of requirements were resolved by #3007 (this is the main PR for this issue I would say)