livebook-dev / livebook

Automate code & data workflows with interactive Elixir notebooks
https://livebook.dev
Apache License 2.0
4.62k stars 408 forks source link

Release cookie generation on start causing problem with systemd #2691

Closed zacksiri closed 1 week ago

zacksiri commented 1 week ago

Environment

Current behavior

I started my livebook using systemd so that when i start my OS livebook automatically starts. Now when I start the app with systemd I notice the following:

● livebook.service - Livebook startup
     Loaded: loaded (/etc/systemd/system/livebook.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-06-28 11:57:02 UTC; 3s ago
   Main PID: 3224 (livebook)
      Tasks: 5 (limit: 76888)
     Memory: 1.7M (peak: 2.6M)
        CPU: 6.178s
     CGroup: /system.slice/livebook.service
             ├─3224 /bin/sh /root/livebook/_build/prod/rel/livebook/bin/livebook daemon
             ├─3233 /bin/sh /root/livebook/_build/prod/rel/livebook/bin/livebook daemon
             ├─3234 cat /dev/urandom
             ├─3235 tr -dc a-zA-Z0-9
             └─3236 fold -w 32

Expected behavior

Livebook starts normally:

● livebook.service - Livebook startup
     Loaded: loaded (/etc/systemd/system/livebook.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-06-28 12:22:07 UTC; 4min 50s ago
   Main PID: 3526 (beam.smp)
      Tasks: 94 (limit: 76888)
     Memory: 103.5M (peak: 111.5M)
        CPU: 2.894s
     CGroup: /system.slice/livebook.service
             ├─3526 /root/.asdf/installs/erlang/26.2.5/erts-14.2.5/bin/beam.smp -sbwt none -sbwtdcpu none -sbwtdio none -Q 65536 -->
             ├─3640 erl_child_setup 1024
             ├─3727 sh -s disksup
             ├─3729 /root/.asdf/installs/erlang/26.2.5/lib/os_mon-2.9.1/priv/bin/memsup
             ├─3730 /root/.asdf/installs/erlang/26.2.5/lib/os_mon-2.9.1/priv/bin/cpu_sup
             ├─3734 /root/.asdf/installs/erlang/26.2.5/erts-14.2.5/bin/inet_gethost 4
             └─3735 /root/.asdf/installs/erlang/26.2.5/erts-14.2.5/bin/inet_gethost 4

Analysis

To get livebook to start normally I had to replace the following line:

in the file: _build/prod/rel/livebook/releases/0.14.0-dev/env.sh there was a line with the following:

export RELEASE_COOKIE="${RELEASE_COOKIE:-$(cat /dev/urandom | env LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)}"

I replaced this with:

export RELEASE_COOKIE="livebook"

and everything works normally.

I noticed that the problem is here: https://github.com/livebook-dev/livebook/blob/main/rel/server/env.sh.eex

Any chance we can change the implementation to be friendly with systemd?

zacksiri commented 1 week ago

So I managed to find the fix for this. In systemd there is a setting called IgnoreSIGPIPE which needs to be set to false and it will resolve this issue.

There is more explanation here: https://stackoverflow.com/questions/44358723/systemd-unit-file-never-finishes-when-using-a-shell-command-with-pipes

Here is an example:

[Unit]
Description=Livebook startup
After=network.target

[Service]
IgnoreSIGPIPE=false
User=root
Group=root
Environment="PATH=/root/.asdf/shims:/root/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
# Environment="LIVEBOOK_IP=0.0.0.0"
Environment="LIVEBOOK_PASSWORD=changeme"
Environment="MIX_ENV=prod"
Environment="LIVEBOOK_CACERTFILE=/etc/ssl/certs/ca-certificates.crt"
# Environment="LIVEBOOK_IDENTITY_PROVIDER=tailscale:/var/run/tailscale/tailscaled.sock"

ExecStart=/root/livebook/_build/prod/rel/livebook/bin/livebook start

[Install]
WantedBy=multi-user.target