Document how to enable serial console tty

davidak commented 4 years ago

I wanted to enable serial console tty in NixOS on a server, so i can still use the terminal even when i messed up the networking config. Sadly, it was not documented in the NixOS Manual how to do it.

So please add documentation for that to NixOS manual.

I found some hints in descriptions of options like systemd.services.<name>.enable:

Note that enable=true does not make a unit start by default at boot; if you want that, see wantedBy.

and systemd.services.<name>.wantedBy:

Units that want (i.e. depend on) this unit. The standard way to make a unit start by default at boot is to set this option to [ "multi-user.target" ].

So the needed configuration looks like this:

  # start tty0 on serial console
  systemd.services."serial-getty@ttyS0" = {
    enable = true;
    wantedBy = [ "getty.target" ]; # to start at boot
    serviceConfig.Restart = "always"; # restart when session is closed
  };

misuzu commented 4 years ago

You can also use this:

{
  boot.kernelParams = [
    "console=ttyS0,115200"
    "console=tty1"
  ];
}

Required systemd unit will start automatically. If you want to see boot log in serial console "console=ttyS0,115200" should be after "console=tty1".

davidak commented 4 years ago

Required systemd unit will start automatically.

Will it get stopped when i close the session, like in the solution above?

I had to explicitely enable restarts, but i think that can get added upstream to the generator template.

serviceConfig.Restart = "always";

misuzu commented 4 years ago

Required systemd unit will start automatically.

Will it get stopped when i close the session, like in the solution above?

I had to explicitely enable restarts, but i think that can get added upstream to the generator template.

serviceConfig.Restart = "always";

Restarts is enabled by default, i didn't change anything.

> systemctl cat serial-getty@ttyS0.service                                                                                                                                       misuzu@serara 10:41:47
# /nix/store/xiq3lj3m6rgkigi6fkqiz28b88qakzcz-systemd-243.7/example/systemd/system/serial-getty@.service
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Serial Getty on %I
Documentation=man:agetty(8) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html
BindsTo=dev-%i.device
After=dev-%i.device systemd-user-sessions.service plymouth-quit-wait.service getty-pre.target

# If additional gettys are spawned during boot then we should make
# sure that this is synchronized before getty.target, even though
# getty.target didn't actually pull it in.
Before=getty.target
IgnoreOnIsolate=yes

# IgnoreOnIsolate causes issues with sulogin, if someone isolates
# rescue.target or starts rescue.service from multi-user.target or
# graphical.target.
Conflicts=rescue.service
Before=rescue.service

[Service]
# The '-o' option value tells agetty to replace 'login' arguments with an
# option to preserve environment (-p), followed by '--' for safety, and then
# the entered username.
ExecStart=-/sbin/agetty -o '-p -- \\u' --keep-baud 115200,38400,9600 %I $TERM
Type=idle
Restart=always
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes

[Install]
WantedBy=getty.target

# /nix/store/gzhzh7vqf5wk9mhn5dcagvl0h496kykw-system-units/serial-getty@.service.d/overrides.conf
[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/k51wcq4lqkj3la51jlpnsnnly5wwhc8i-glibc-locales-2.30/lib/locale/locale-archive"
Environment="PATH=/nix/store/hzvl3dvv8651iqlb5g6gq5hzjzmhjn7m-coreutils-8.31/bin:/nix/store/rphcpivxfm4blw36ki262yrmqygw9pcp-findutils-4.7.0/bin:/nix/store/pmwf1fz03c5say5fnqk4m748rjb8kdy7-gnugrep-3.4/bin:/nix/store/9k90a16frvqdcxzkhxcjsi2>
Environment="TZDIR=/nix/store/msbzkf26r6v5nypj1dspjk5jfnvl9l8y-tzdata-2019c/share/zoneinfo"

X-RestartIfChanged=false

ExecStart=
ExecStart=@/nix/store/hw29x9k79b7m7gmd1fks72h1n5hk0ivh-util-linux-2.33.2-bin/sbin/agetty agetty --login-program /nix/store/s30n68sarsjsclb9l8risvq9zvg25gg1-shadow-4.8/bin/login  %I 115200,57600,38400,9600 $TERM

flokli commented 4 years ago

You should only need to enable serial consoles by adding them to the kernel cmdline as described in https://github.com/NixOS/nixpkgs/issues/84105#issuecomment-608084218.

systemd will automatically create serial-getty@….service symlinks while parsing the kernel cmdline - see https://github.com/systemd/systemd/blob/master/src/getty-generator/getty-generator.c.

davidak commented 4 years ago

I have bootet from usb, but it seems the minimal image also don't starts a serial console by default. It would help me right now. Would that has any negative effect when the hardware has no serial port, like failed systemd unit?

We could also detect serial console hardware from nixos-generate-config and activate it when available? My colluege said it works on other systems by default, like debian.

misuzu commented 4 years ago

Yeah, enabled serial console on installation cd/usb/netboot would be great!

flokli commented 4 years ago

We might want to look at https://github.com/NixOS/nixpkgs/issues/30760#issuecomment-416093820 and the references there.

devhell commented 4 years ago

You can also use this:
{
  boot.kernelParams = [
    "console=ttyS0,115200"
    "console=tty1"
  ];
}
Required systemd unit will start automatically. If you want to see boot log in serial console "console=ttyS0,115200" should be after "console=tty1".

This doesn't work for me. I'm running a NixOS VM via virsh and this would be needed for virsh console nixosvm to work. The need arises from the VM also having LUKS active. So, while I could use virt-viewer to enter the LUKS passphrase, I think using virsh console would be more straightforward, but for that the serial-getty@ttys0.service needs to start on the VM, but the systemd unit is not generated even though I have the above suggested in the VM's configuration.nix. The VM is running 20.03.

flokli commented 4 years ago

We're mixing two things here.

The initial issue is about serial console access for a running system. That can be archieved by adding boot.kernelParams as described in https://github.com/NixOS/nixpkgs/issues/84105#issuecomment-608084218. Entering crypto keys during early boot is something different - we currently don't have systemd in initramfs yet, and gettys are spun up much later. However, this should solve the initial issue, recovering from breaking the network configuration (which should probably be added to the manual, if it's not already).

The current existing tooling can currently only ask on one console (and which one might depend on the order). I'd consider asking on the graphical console a sane default if there's a graphical output. I didn't yet debug on how these options need to be set if you want to prefer the serial console for this question. @devhell if you want to do some experiments and contribute that to the documentation, that'd be appreciated!

Getting systemd into initramfs, and building systemd with cryptsetup support would allow asking for the crypto volume password using the systemd-ask-password mechanism, which should ask on all serial consoles - so just adding the consoles to boot.kernelParams should work some time in the future - but it'll still take some time till then.

devhell commented 4 years ago

Thanks @flokli. I'm sorry for mixing those two things up. I'll elaborate a bit. I was under the impression that adding boot.kernelParams to the VM's configuration would allow me to execute virsh console on the host to access the VM's console and enter the LUKS passphrase to allow the VM to finish its boot. After I added boot.kernelParams I rebuilt the VM and rebooted it, virsh console would not give me a console into the VM. So I entered the passphrase via virt-viewer and checked virsh console but I would still not get anything back from the console once the boot completed. I tried to verify via systemctl that serial-getty@ttys0 had been generated, but it had not.

I then used @davidak's solution instead and added systemd.services."serial-getty@ttyS0" to the VM's configuration, rebuild, reboot, and still no virsh console to get to the LUKS passphrase input, but after using virt-viewer to enter the passphrase virsh console started showing the rest of the boot process and subsequent login prompt, etc. Checking for serial-getty@ttys0 was also successful.

I then checked if enabling GRUB's early console activation would work as stated in 20.03's manual, using:

boot.loader.grub.extraConfig = ''
      serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
      terminal_input --append serial
      terminal_output --append serial
'';

This worked and using virsh console I was able to see the GRUB boot menu and then the LUKS passphrase prompt. I then entered the passphrase and the boot continued as expected and shortly after a login prompt was available on the console.

Just to test things out, I've then removed the explicit addition of systemd.services."serial-getty@ttys0", rebuild, rebooted, and confirmed that only the GRUB menu was visible via virsh console and no LUKS passphrase prompt. I then added the original solution via boot.kernelParams, rebuild, rebooted, and now I suddenly also got the LUKS passphrase prompt on the console. So I checked if serial-getty@ttys0 was generated, and now it was.

Finally, I removed GRUB's extraConfig so that only boot.kernelParams was left over, rebuild, rebooted, and now virsh console gives me no GRUB menu, but does give me a LUKS prompt.

I don't know why it's behaving like this, but I hope someone might come up with an idea and finds this write-up useful.

Thank you!

flokli commented 4 years ago

@devhell with just the boot.kernelParams being set, did you check with systemctl status if there was any getty spawned on the serial console?

devhell commented 4 years ago

@flokli, the first time I set boot.kernelParams nothing was spawned, then I added systemd.service."serial-getty@ttys0" and getty was spawned as I would have expected using boot.kernelParams. However, after removing systemd.service."serial-getty@ttys0", rebuilding and rebooting, the getty was still being spawned, so I'm not sure why it wasn't doing it the first time around before I added systemd.service."serial-getty@ttys0".

flokli commented 4 years ago

Can you provide a minimal configuration snippet producing an image that I can debug with libvirt?

This "works after some reboots" behaviour is something I'd really like to take a closer look at.

devhell commented 4 years ago

Here you go, I hope this is what you had in mind: https://gist.github.com/devhell/d23b0d09470894060645ad13b35e4c3c

The VM is LUKS encrypted, nothing fancy, example here, with UUIDs removed: https://gist.github.com/devhell/07df7d1902ae2ab4af6456a8c6e0989a

Hope that helps and thank you for looking into this.

devhell commented 4 years ago

Heya @flokli, just wondered if you had been able to test this?

flokli commented 4 years ago

Hey! Sorry for the silence - didn't get to this until now.

I slightly updated your configuration, and used the build-qcow2.nix to build a nixos.qcow2 image:

nix-build '<nixpkgs/nixos>' -A config.system.build.qcow2 --arg configuration "{ imports = [ ./build-qcow2.nix ]; }" -I nixpkgs=/path/to/my/nixpkgs

I copied nixos.qcow2 from the resulting store path, and used "Import existing disk image" from virt-manager. I selected "Generic default" as Operating system, and manually increased memory and cpu, but selecting "NixOS Unstable" also seemed to have the same effect.

I could both see and update Grub counting down at View -> Text Consoles -> {Serial 1, Graphical Console Spice}.

Without console=tty0 (as in your example) at the beginning of boot.kernelParams, I saw all Linux' boot output (including that from stage1/2, which would ask for the luks passphrase) at the serial console.
If I added console=tty0 at the front, that output appeared on the graphical console instead.

In both cases, getty did start up on both consoles after boot was finished, and I didn't experience different behaviour across reboots as described in https://github.com/NixOS/nixpkgs/issues/84105#issuecomment-654844464.

stage1/2 seems to only be able to ask for a password on one console.

If I removed console=ttyS0… from boot.kernelParams, there was no output during linux booting, and no getty was spawned there (which makes sense, given systemd starts gettys on all the ttys, and parses the kernel cmdline for additional consoles).

TLDR:

Grub can be used simultaneously from serial(s) and graphical consoles, using the boot.loader.grub.extraConfig from above
systemd will spin up gettys (and spin them up again if you exit your session) for all virtual terminals and consoles passed via boot.kernelParams (see http://0pointer.de/blog/projects/serial-console.html for details)
stage1/2 can only ask for a password on one console, it'll pick the first one from boot.kernelParams.

@devhell can you reproduce this?

devhell commented 4 years ago

Hey @flokli,

Thank you very much for testing this. Yes, I can reproduce it, and now it also makes sense. :)

Much appreciated!

flokli commented 4 years ago

Awesome. If you can put this somewhere into the Wiki, this'd be :heart: :wink:

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

goertzenator commented 2 years ago

Hi, has there been any updates on starting a serial getty?

I am not getting a serial getty despite having this:

    boot.kernelParams = [ "console=ttyS0,115200" "console=tty0" ];

But in any case, I would like to have additional serial gettys on serial ports that are not included in the above kernelParams. Options?

Some context:

I am generating an .iso image that needs these serial capabilities.
An embedded device will boot this iso image for software installation. It does have video/usb, but we'd like to be able to use a serial port too.

EDIT: I just spotted the right bits above, so I am set!

    systemd.services."serial-getty@ttyS0" = {
      enable = true;
      wantedBy = [ "getty.target" ]; # to start at boot
      serviceConfig.Restart = "always"; # restart when session is closed
    };

    systemd.services."serial-getty@ttyS2" = {
      enable = true;
      wantedBy = [ "getty.target" ]; # to start at boot
      serviceConfig.Restart = "always"; # restart when session is closed
    };

19ngaddam commented 1 year ago

Hey, I'm trying to create VMs using nix. I've followed these steps and added: boot.kernelParams = [ "console=ttyS0,115200" "console=tty0" ]; . However, when I try to access a vm via virsh console I am unable to interact with the VM and instead get dropped into a console with logs from systemd units. Is there a way to get an interactive console? Unfortunately, I don't have access to any GUI and networking isn't set up.

Any help would be appreciated!

Mic92 commented 1 year ago

Hey, I'm trying to create VMs using nix. I've followed these steps and added: boot.kernelParams = [ "console=ttyS0,115200" "console=tty0" ]; . However, when I try to access a vm via virsh console I am unable to interact with the VM and instead get dropped into a console with logs from systemd units. Is there a way to get an interactive console? Unfortunately, I don't have access to any GUI and networking isn't set up.

Any help would be appreciated!

You need to figure out what tty "virsh console" is actually using and make sure a getty is started on that one.

Mic92 commented 1 year ago

Those defaults work quite well for servers and virtual machines: https://github.com/numtide/srvos/blob/main/nixos/common/serial.nix They are inspired by ubuntu and alpine + some nicer defaults to set terminal size and defaults on serial consoles

useche commented 1 month ago

For some reason, simply adding the console kernel parameter for my raspberry pi didn't work. I'm not sure why, but systemd didn't start the serial-getty service on its own. I was able to enable the GPIO serial console using the following configuration:

  # Enables serial console through the GPIO.
  boot.kernelParams = [ "console=ttyS1,115200" ];
  systemd.services."serial-getty@ttyS1" = {
    enable = true;
    serviceConfig.Restart = "always";

  };

Note how I use ttyS1 instead of ttyS0.

flokli commented 1 month ago

I've been running into these issues as well. I think the direction these days is to not set any console-related kernel cmdline options, so the device tree defaults can take over.

I can send a PR for this, but we need to test it doesn't break console on some common boards that might be missing it.

NixOS / nixpkgs

Document how to enable serial console tty #84105