Access guest console when it is running

[praveenkumar]

As of now we https://github.com/crc-org/vfkit/blob/c9a4b0861530ce7320aae34ad2394f41e2fd5e83/pkg/vf/virtio.go#L272-L276 connects the VM serial output to the stdin/stdout of the terminal where vfkit is running but there is no way to connect to it so that we can access VM console even in case of no ssh connectivity.

https://developer.apple.com/documentation/virtualization/vzfilehandleserialportattachment?language=objc => this api allows bidirectional communication using file handles but we don't know how to consume it.

This is important for crc side to debug some of the issues when ssh connection is broken.

[cfergeau]

hyperkit has such support, I think the code they use for this is https://github.com/moby/hyperkit/blob/3cb0d5475244761c61e6fd4a562702c26b46f846/src/lib/uart_emul.c#L738-L784

The code for https://pkg.go.dev/github.com/pkg/term/termios#Pty is very similar: https://github.com/pkg/term/blob/v1.1.0/termios/pty.go#L22

[cfergeau]

Seems to be relatively easy to do https://github.com/cfergeau/vfkit/tree/pty This is a hack which changes --device virtio-serial,stdio to output to a pseudoterminal instead of stdio. Need to add cmdline parsing/... to this proof of concept.

[cdrage]

Testing with https://github.com/cfergeau/vfkit/tree/pty I made some awesome progress on integrating some of this within PD!

I did encounter some issues however:

• Running the same command twice on a different image, I would get a different ttys which is to be expected.. but I couldn't access it / read / write.

Start VM 1

./vfkit-arm64 --cpus 2 --memory 2048 \
    --bootloader efi,variable-store=./efi-variable-store,create \
    --device virtio-blk,path=/Users/cdrage/bootc/image/disk.raw \
    --device virtio-serial,pty \
    --device virtio-net,nat,mac=72:20:43:d4:38:62 \
    --device virtio-rng \
    --device virtio-input,keyboard \
    --device virtio-input,pointing

Start VM 2

./vfkit-arm64 --cpus 2 --memory 2048 \
    --bootloader efi,variable-store=./efi-variable-store,create \
    --device virtio-blk,path=/Users/cdrage/bootc/image/disk2.raw \
    --device virtio-serial,pty \
    --device virtio-net,nat,mac=72:20:43:d4:38:62 \
    --device virtio-rng \
    --device virtio-input,keyboard \
    --device virtio-input,pointing

You should get two ttys.

One is /dev/ttys003

The other would be random (mine was /dev/ttys010).

If you do screen /dev/ttys003 there is no issue.

If you do screen /dev/ttys010 you get a permission error.

Video is below!

https://github.com/crc-org/vfkit/assets/6422176/523d8df0-167b-4826-b95c-26e442cf53da

• It would be nice to specify exactly which path the ttys will go (be able to pick /dev/ttys123, etc.)

• Unable to see boot screen / GRUB, only was able to see the login prompt.

[cfergeau]

You can remove

    --device virtio-input,keyboard \
    --device virtio-input,pointing

from your commandline, they are only needed if you use the --gui flag.

[cdrage]

You can remove

    --device virtio-input,keyboard \
    --device virtio-input,pointing

from your commandline, they are only needed if you use the --gui flag.

Thanks! I shouldn't of added those inputs haha.

I did try it again without those inputs and I can now "load" the screen, but I get a different error. It's now completely blank / wont load anything.

https://github.com/crc-org/vfkit/assets/6422176/0a3b64a4-8519-405a-b505-ccf4ba321e5e

[cfergeau]

* It would be nice to specify exactly which path the ttys will go (be able to pick /dev/ttys123, etc.)

This is one issue with the current branch, there is no nice way of providing the pty name to vfkit users. It's printed in the logs and that's it. I'm not sure it's possible to choose which tty will be allocated, I'll need to look more closely.

hyperkit allows to specify a file when using the autopty option, but this is the path to a file where the pty name will be written https://github.com/moby/hyperkit/blob/3cb0d5475244761c61e6fd4a562702c26b46f846/src/lib/uart_emul.c#L767-L779 (ie autopty=/tmp/foo will write /dev/ttyS123 to /tmp/foo if this is the pty which was allocated).

I hope to be able to give this info through vfkit REST API, calling /inspect would return a json string which would have the pty name used among a lot of other info.

[cfergeau]

I'm not sure it's possible to choose which tty will be allocated, I'll need to look more closely.

The low-level kernel/libc API is definitely of the form "allocate a pty for me, and give me its name", it does not seem possibe to control this and choose a name. However https://github.com/cfergeau/vfkit/tree/pty now exposes the pty name in the REST API

% curl --unix-socket ~/dev/vfkit/rest.sock "http://vfkit/vm/inspect" |gojq 
{
[snip]
  "devices": [
    {
      "devName": "virtio-blk",
      "imagePath": "overlay.img",
      "kind": "virtioblk"
    },
    {
      "kind": "virtioserial",
      "ptyName": "/dev/ttys005",
      "usesPty": true
    },
    {
      "kind": "virtionet",
      "macAddress": "72:20:43:d4:38:62",
      "nat": true
    },
    {
      "kind": "virtiorng"
    }
  ],
  "memoryBytes": 2147483648,
  "vcpus": 2
}

[cfergeau]

Running the same command twice on a different image, I would get a different ttys which is to be expected.. but I couldn't access it / read / write.

I have been investigating a bunch of similar issues, sometimes even after killing/starting again the same VM, I can't connect. My current suspicion is that it's related to my non-handling of Close() on the PTY sockets. They are leaked, and go closes them as it sees fit when garbage collection happens. If I assign them to global variables so that they are not garbage collected, behaviour is more predictible so far (I can kill/start/kill/start with no problems, I can run 2 VMs side by side and connect to both, ..)

[cfergeau]

Conclusion of the investigations is that it was all related, closing the master file descriptor too early will remove the ttysxxx device which was allocated, and will reset its permissions to root.root. https://jalopezg.dev/2023/11/14/UNIX98-pts/ The latest iterations of this PR keep the master file descriptor open as long as vfkit runs.

[gbraad]

Are you waiting for peer-review/confirmation? If so, from whom?

[cfergeau]

Are you waiting for peer-review/confirmation? If so, from whom?

The associated PR will need a review https://github.com/crc-org/vfkit/pull/113