search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

LXD random container "hangs" on startup/shutdown #11228

Closed ewildgoose closed 1 year ago

ewildgoose commented 1 year ago

Using Gentoo/openrc on an older Intel E3. Kernel 6.0.12

Symptoms are that various types of container image will stop responding on either startup or shutdown. However, once started or stopped they behave as expected. Phrased another way, if the instance hangs at startup, then "killing it" (see below) and restarting it will usually succeed the second time, it will then behave as expected with all networking functioning as expected, etc. When it hangs on startup it appears to run through normal bootup and hangs at the end of normal startup. Console will not respond

Similarly it will usually normally shutdown ok, but sometimes it will reach the end of the normal shutdown (ie console debug output looks same as always) and then hang with the init process showing "defunct". Killing the monitor process and it can then usually be restarted and stopped as expected

It doesn't seem to be possible to terminate the stuck process using lxc tooling. ie "lxc stop stuckcontainer -f --timeout 2" will just sit and hang. Sending kill -9 to the monitor process will cause it to terminate (kill without flags will not)

I can't find a common trigger. However, It feels like a race, as it seems more likely to occur if I start several instances simultaneously. It might be more likely to trigger a hang if I add various network devices to the container as well (hard to be sure - it feels to occur more frequently). It is entirely possible that I don't have cgroups correctly configured, although I can't find any log messages implying this

Here is a repro:

sm3 ~ # lxc launch images:ubuntu/jammy v1
Creating v1

The instance you are starting doesn't have any network attached to it.
  To create a new network, use: lxc network create
  To attach a network to an instance, use: lxc network attach

Starting v1
sm3 ~ # lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
alive
sm3 ~ # lxc stop v1 --force --timeout 2
sm3 ~ # lxc start v1 && sleep 1 && lxc exec v1 -- echo alive
Error: Instance is not running
sm3 ~ #
sm3 ~ # ps ax|grep v1
 8519 ?        Ss     0:00 [lxc monitor] /var/lib/lxd/containers v1
 8676 pts/3    S+     0:00 grep --colour=auto v1
sm3 ~ # lxc exec v1 -- echo alive
Error: Instance is not running
sm3 ~ #
sm3 ~ # tail /var/log/lxd/v1/console.log -n 100
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[UNSUPP] Starting of Arbitrary Executable Fi…ystem Automount Point unsupported.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Path Units.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slice Units.
[  OK  ] Reached target Swaps.
[  OK  ] Reached target Local Verity Protected Volumes.
[  OK  ] Listening on Syslog Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Reached target Socket Units.
systemd-journald.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/unified/system.slice/systemd-journald.service failed: Invalid argument
         Starting Journal Service...
         Starting Set the console keyboard layout...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module drm...
         Starting Load Kernel Module fuse...
         Starting Generate network units from Kernel command line...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
         Starting Coldplug All udev Devices...
modprobe@configfs.service: Deactivated successfully.
[  OK  ] Finished Load Kernel Module configfs.
modprobe@drm.service: Deactivated successfully.
[  OK  ] Finished Load Kernel Module drm.
modprobe@fuse.service: Deactivated successfully.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished Generate network units from Kernel command line.
[  OK  ] Finished Remount Root and Kernel File Systems.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Reached target Preparation for Network.
         Starting Create System Users...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Set the console keyboard layout.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
[  OK  ] Reached target Local File Systems.
         Starting Set console font and keymap...
         Starting Rule-based Manager for Device Events and Files...
[  OK  ] Finished Set console font and keymap.
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Rule-based Manager for Device Events and Files.
         Starting Network Configuration...
[  OK  ] Finished Create Volatile Files and Directories.
[  OK  ] Reached target System Time Set.
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Started Network Configuration.
         Starting Network Name Resolution...
[  OK  ] Finished Coldplug All udev Devices.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Daily dpkg database backup timer.
[  OK  ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Message of the Day.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started Ubuntu Advantage Timer for running repeated jobs.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Timer Units.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Started Regular background program processing daemon.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Save initial kernel messages after boot.
         Starting Remove Stale Online ext4 Metadata Check Snapshots...
         Starting Dispatcher daemon for systemd-networkd...
         Starting System Logging Service...
         Starting User Login Management...
[  OK  ] Started System Logging Service.
[  OK  ] Finished Remove Stale Online ext4 Metadata Check Snapshots.
[  OK  ] Started User Login Management.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Network.
[  OK  ] Reached target Host and Network Name Lookups.
         Starting Permit User Sessions...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Created slice Slice /system/getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Record Runlevel Change in UTMP...
[  OK  ] Finished Record Runlevel Change in UTMP.

Ubuntu 22.04.1 LTS v1 console

v1 login: sm3 ~ #

The last few lines of logging from lxd are (in the next comment)

I am then unable to stop the instance using lxc tooling:

sm3 ~ # lxc stop v1 --force --timeout 2
Error: The instance is already stopped
Try `lxc info --show-log v1` for more info

sm3 ~ # tail /var/log/lxd/v1/console.log
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Record Runlevel Change in UTMP...
[  OK  ] Finished Record Runlevel Change in UTMP.

Ubuntu 22.04.1 LTS v1 console

v1 login:

sm3 ~ # ps -ef |grep 8519
root      8519     1  0 11:34 ?        00:00:00 [lxc monitor] /var/lib/lxd/containers v1
1000000   8527  8519  0 11:34 ?        00:00:00 /sbin/init
sm3 ~ # kill 8527
sm3 ~ # kill 8519
sm3 ~ # ps -ef |grep 8519
root      8519     1  0 11:34 ?        00:00:00 [lxc monitor] /var/lib/lxd/containers v1
1000000   8527  8519  0 11:34 ?        00:00:00 /lib/systemd/systemd --system --deserialize 13
root      8909 19597  0 11:45 pts/3    00:00:00 grep --colour=auto 8519
sm3 ~ # ps -ef |grep 8519
root      8519     1  0 11:34 ?        00:00:00 [lxc monitor] /var/lib/lxd/containers v1
1000000   8527  8519  0 11:34 ?        00:00:00 /lib/systemd/systemd --system --deserialize 13
root      8912 19597  0 11:45 pts/3    00:00:00 grep --colour=auto 8519
sm3 ~ # kill 8519
sm3 ~ # kill 8519
sm3 ~ # kill -9 8519
sm3 ~ # ps -ef |grep 8519
root      8997 19597  0 11:45 pts/3    00:00:00 grep --colour=auto 8519
sm3 ~ # ps -ef |grep mon
root      8999 19597  0 11:45 pts/3    00:00:00 grep --colour=auto mon

So I can force kill the monitor task and it appears to close down the instance (it leaves the devices mounted).

While the container is in this stuck state I can use lxc tooling to start/stop other containers, but lxc list simply hangs forever without returning results. This makes it difficult to understand which instance is stuck and force restart it. eg consider

sm3 ~ # lxc start gentoo1 v1 gitea
sm3 ~ # lxc list status=RUNNING
+---------+---------+------------------------+------------------------------------------------+-----------+-----------+
|  NAME   |  STATE  |          IPV4          |                      IPV6                      |   TYPE    | SNAPSHOTS |
+---------+---------+------------------------+------------------------------------------------+-----------+-----------+
| gentoo1 | RUNNING | 192.168.242.88 (priv0) | fd42:6980:7ca3:9f8a:68:9391:fdac:b652 (priv0)  | CONTAINER | 0         |
|         |         |                        | fd42:25e7:9b4f:f5e6:216:3eff:fe3e:648c (nat0)  |           |           |
+---------+---------+------------------------+------------------------------------------------+-----------+-----------+
| gitea   | RUNNING | 91.220.24.154 (net0)   | fd42:6980:7ca3:9f8a:216:3eff:fe75:9051 (priv0) | CONTAINER | 1         |
|         |         | 192.168.242.63 (priv0) |                                                |           |           |
+---------+---------+------------------------+------------------------------------------------+-----------+-----------+
| v1      | RUNNING |                        |                                                | CONTAINER | 0         |
+---------+---------+------------------------+------------------------------------------------+-----------+-----------+
sm3 ~ # lxc stop gentoo1 v1 gitea

... hangs here forever...

From this state I can't easily figure out which of the instances hung on shutdown, other than using ps. FWIW, in that specific example it was the gitea instance and the console log for that shows the expected last few lines, identically to when it doesn't hang

I've just run the example with the v1 instance a few more times and after a few more successful start/stop cycles it then hung on shutdown (vs startup). So I don't see a pattern that it's systemd/openrc nor startup/shutdown as the specific common theme

The only clue I see is that the hang on startup is not immediate, but after a few seconds. So consider the following where I repeated starting and stopping that same v1 instance. This time I add --console to the start line.

sm3 ~ # lxc start v1 --console
To detach from the console, press: <ctrl>+a q
Queued start job for default target Graphical Interface.
[  OK  ] Created slice Slice /system/modprobe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[UNSUPP] Starting of Arbitrary Executable Fi…ystem Automount Point unsupported.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Path Units.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slice Units.
[  OK  ] Reached target Swaps.
[  OK  ] Reached target Local Verity Protected Volumes.
[  OK  ] Listening on Syslog Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Reached target Socket Units.
systemd-journald.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/unified/system.slice/systemd-journald.service failed: Invalid argument
         Starting Journal Service...
         Starting Set the console keyboard layout...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module drm...
         Starting Load Kernel Module fuse...
         Starting Generate network units from Kernel command line...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
         Starting Coldplug All udev Devices...
modprobe@configfs.service: Deactivated successfully.
[  OK  ] Finished Load Kernel Module configfs.
modprobe@drm.service: Deactivated successfully.
[  OK  ] Finished Load Kernel Module drm.
modprobe@fuse.service: Deactivated successfully.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished Generate network units from Kernel command line.
[  OK  ] Finished Remount Root and Kernel File Systems.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Reached target Preparation for Network.
         Starting Create System Users...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Set the console keyboard layout.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
[  OK  ] Reached target Local File Systems.
         Starting Set console font and keymap...
         Starting Rule-based Manager for Device Events and Files...
[  OK  ] Finished Set console font and keymap.
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Rule-based Manager for Device Events and Files.
         Starting Network Configuration...
[  OK  ] Finished Create Volatile Files and Directories.
[  OK  ] Reached target System Time Set.
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Started Network Configuration.
         Starting Network Name Resolution...
[  OK  ] Finished Coldplug All udev Devices.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Daily dpkg database backup timer.
[  OK  ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Message of the Day.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started Ubuntu Advantage Timer for running repeated jobs.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Timer Units.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Started Regular background program processing daemon.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Save initial kernel messages after boot.
         Starting Remove Stale Online ext4 Metadata Check Snapshots...
         Starting Dispatcher daemon for systemd-networkd...
         Starting System Logging Service...
         Starting User Login Management...
[  OK  ] Started System Logging Service.
[  OK  ] Finished Remove Stale Online ext4 Metadata Check Snapshots.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Network.
[  OK  ] Reached target Host and Network Name Lookups.
         Starting Permit User Sessions...
[  OK  ] Started User Login Management.
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Created slice Slice /system/getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Record Runlevel Change in UTMP...
[  OK  ] Finished Record Runlevel Change in UTMP.

Ubuntu 22.04.1 LTS v1 console

v1 login:
v1 login:
v1 login:

What you see at the end is that I was able to just press return a few times, then after 1-2 seconds the container stops responding and I cannot get any response through console or lxc exec (or to kill it). However, if it does start up ok, then I see containers running for many hours without trouble (can't yet say "days")

Where could I look for clues on what is happening? (and how to resolve)

ewildgoose commented 1 year ago

lxd log relating to the above

time="2022-12-15T11:33:59Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=root
time="2022-12-15T11:33:59Z" level=debug msg="Event listener server handler started" id=d7961485-c875-4ec6-a925-debd5034d3c6 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:33:59Z" level=debug msg="Handling API request" ip=@ method=POST protocol=unix url=/1.0/instances/v1/exec username=root
time="2022-12-15T11:33:59Z" level=debug msg="API Request\n\t{\n\t\t\"command\": [\n\t\t\t\"echo\",\n\t\t\t\"alive\"\n\t\t],\n\t\t\"wait-for-websocket\": true,\n\t\t\"interactive\": true,\n\t\t\"environment\": {\n\t\t\t\"TERM\": \"tmux-256color\"\n\t\t},\n\t\t\"width\": 245,\n\t\t\"height\": 60,\n\t\t\"record-output\": false,\n\t\t\"user\": 0,\n\t\t\"group\": 0,\n\t\t\"cwd\": \"\"\n\t}" ip=@ method=POST protocol=unix url=/1.0/instances/v1/exec username=root
time="2022-12-15T11:33:59Z" level=debug msg="New operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Started operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/7daf5186-ea3c-4e15-a77d-9eb86648858a\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"7daf5186-ea3c-4e15-a77d-9eb86648858a\",\n\t\t\t\"class\": \"websocket\",\n\t\t\t\"description\": \"Executing command\",\n\t\t\t\"created_at\": \"2022-12-15T11:33:59.943609756Z\",\n\t\t\t\"updated_at\": \"2022-12-15T11:33:59.943609756Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/v1\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/v1\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": {\n\t\t\t\t\"command\": [\n\t\t\t\t\t\"echo\",\n\t\t\t\t\t\"alive\"\n\t\t\t\t],\n\t\t\t\t\"environment\": {\n\t\t\t\t\t\"HOME\": \"/root\",\n\t\t\t\t\t\"LANG\": \"C.UTF-8\",\n\t\t\t\t\t\"PATH\": \"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n\t\t\t\t\t\"TERM\": \"tmux-256color\",\n\t\t\t\t\t\"USER\": \"root\"\n\t\t\t\t},\n\t\t\t\t\"fds\": {\n\t\t\t\t\t\"0\": \"fb183d60124bd078f062a868d27f8af0df4da443bc7424b061d6e2837c867ea5\",\n\t\t\t\t\t\"control\": \"ff42399f7a56557d16db9a2b9e35fa9d1d8c7effd4dbf5ce2488cb0cc93717dd\"\n\t\t\t\t},\n\t\t\t\t\"interactive\": true\n\t\t\t},\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=202
time="2022-12-15T11:33:59Z" level=debug msg="Waiting for exec websockets to connect"
time="2022-12-15T11:33:59Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/operations/7daf5186-ea3c-4e15-a77d-9eb86648858a/websocket?secret=ff42399f7a56557d16db9a2b9e35fa9d1d8c7effd4dbf5ce2488cb0cc93717dd" username=root
time="2022-12-15T11:33:59Z" level=debug msg="Connecting to operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Connected to operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/operations/7daf5186-ea3c-4e15-a77d-9eb86648858a/websocket?secret=fb183d60124bd078f062a868d27f8af0df4da443bc7424b061d6e2837c867ea5" username=root
time="2022-12-15T11:33:59Z" level=debug msg="Connecting to operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Connected to operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/operations/7daf5186-ea3c-4e15-a77d-9eb86648858a username=root
time="2022-12-15T11:33:59Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"7daf5186-ea3c-4e15-a77d-9eb86648858a\",\n\t\t\t\"class\": \"websocket\",\n\t\t\t\"description\": \"Executing command\",\n\t\t\t\"created_at\": \"2022-12-15T11:33:59.943609756Z\",\n\t\t\t\"updated_at\": \"2022-12-15T11:33:59.943609756Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/v1\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/v1\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": {\n\t\t\t\t\"command\": [\n\t\t\t\t\t\"echo\",\n\t\t\t\t\t\"alive\"\n\t\t\t\t],\n\t\t\t\t\"environment\": {\n\t\t\t\t\t\"HOME\": \"/root\",\n\t\t\t\t\t\"LANG\": \"C.UTF-8\",\n\t\t\t\t\t\"PATH\": \"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n\t\t\t\t\t\"TERM\": \"tmux-256color\",\n\t\t\t\t\t\"USER\": \"root\"\n\t\t\t\t},\n\t\t\t\t\"fds\": {\n\t\t\t\t\t\"0\": \"fb183d60124bd078f062a868d27f8af0df4da443bc7424b061d6e2837c867ea5\",\n\t\t\t\t\t\"control\": \"ff42399f7a56557d16db9a2b9e35fa9d1d8c7effd4dbf5ce2488cb0cc93717dd\"\n\t\t\t\t},\n\t\t\t\t\"interactive\": true\n\t\t\t},\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:33:59Z" level=debug msg="Retrieved PID of executing child process" attachedPid=8474 instance=v1 instanceType=container project=default
time="2022-12-15T11:33:59Z" level=debug msg="Instance process started" PID=8474 instance=v1 interactive=true project=default
time="2022-12-15T11:33:59Z" level=debug msg="Exec control handler started" PID=8474 instance=v1 interactive=true project=default
time="2022-12-15T11:33:59Z" level=debug msg="Exec mirror websocket started" PID=8474 instance=v1 interactive=true number=0 project=default
time="2022-12-15T11:33:59Z" level=debug msg="Instance process stopped" PID=8474 err="<nil>" exitStatus=0 instance=v1 interactive=true project=default
time="2022-12-15T11:33:59Z" level=debug msg="Exec control handler finished" PID=8474 instance=v1 interactive=true project=default
time="2022-12-15T11:33:59Z" level=debug msg="Detected poll(POLLHUP) event: exiting"
time="2022-12-15T11:33:59Z" level=debug msg="Sending write barrier"
time="2022-12-15T11:33:59Z" level=debug msg="DefaultWriter got error getting next reader" err="websocket: close 1006 (abnormal closure): unexpected EOF"
time="2022-12-15T11:33:59Z" level=debug msg="Exec mirror websocket finished" PID=8474 instance=v1 interactive=true number=0 project=default
time="2022-12-15T11:33:59Z" level=debug msg="Updated metadata for operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Success for operation" class=websocket description="Executing command" operation=7daf5186-ea3c-4e15-a77d-9eb86648858a project=default
time="2022-12-15T11:33:59Z" level=debug msg="Event listener server handler stopped" listener=d7961485-c875-4ec6-a925-debd5034d3c6 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:01Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=root
time="2022-12-15T11:34:01Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"192.168.2.12\",\n\t\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\",\n\t\t\t\t\"acme\",\n\t\t\t\t\"internal_metrics\",\n\t\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\t\"remote_token_expiry\",\n\t\t\t\t\"init_preseed\",\n\t\t\t\t\"storage_volumes_created_at\",\n\t\t\t\t\"cpu_hotplug\",\n\t\t\t\t\"projects_networks_zones\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"192.168.2.12:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw\\nHhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51\\neGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG\\nBSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP\\nfViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da\\np3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF\\nBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA\\nAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ\\nlAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO\\noaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"6.0.12-sm\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"2.9\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\": \"sm3\",\n\t\t\t\t\"server_pid\": 31569,\n\t\t\t\t\"server_version\": \"5.9\",\n\t\t\t\t\"storage\": \"lvm\",\n\t\t\t\t\"storage_version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"6.0.2\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:01Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=root
time="2022-12-15T11:34:01Z" level=debug msg="Event listener server handler started" id=24bfffb1-5c0a-4f72-9228-e79252bbd783 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:01Z" level=debug msg="Handling API request" ip=@ method=PUT protocol=unix url=/1.0/instances/v1/state username=root
time="2022-12-15T11:34:01Z" level=debug msg="API Request\n\t{\n\t\t\"action\": \"stop\",\n\t\t\"timeout\": 2,\n\t\t\"force\": true,\n\t\t\"stateful\": false\n\t}" ip=@ method=PUT protocol=unix url=/1.0/instances/v1/state username=root
time="2022-12-15T11:34:01Z" level=debug msg="New operation" class=task description="Stopping instance" operation=d38077d6-bda3-4a0b-8f1e-5e1b6d712a14 project=default
time="2022-12-15T11:34:01Z" level=debug msg="Started operation" class=task description="Stopping instance" operation=d38077d6-bda3-4a0b-8f1e-5e1b6d712a14 project=default
time="2022-12-15T11:34:01Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/d38077d6-bda3-4a0b-8f1e-5e1b6d712a14\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"d38077d6-bda3-4a0b-8f1e-5e1b6d712a14\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Stopping instance\",\n\t\t\t\"created_at\": \"2022-12-15T11:34:01.127095932Z\",\n\t\t\t\"updated_at\": \"2022-12-15T11:34:01.127095932Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/v1\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=202
time="2022-12-15T11:34:01Z" level=debug msg="Stop started" instance=v1 instanceType=container project=default stateful=false
time="2022-12-15T11:34:01Z" level=debug msg="Instance operation lock created" action=stop instance=v1 project=default reusable=false
time="2022-12-15T11:34:01Z" level=info msg="Stopping container" action=stop created="2022-12-15 11:32:27.636648308 +0000 UTC" ephemeral=false instance=v1 instanceType=container project=default stateful=false used="2022-12-15 11:33:58.874331931 +0000 UTC"
time="2022-12-15T11:34:01Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/operations/d38077d6-bda3-4a0b-8f1e-5e1b6d712a14 username=root
time="2022-12-15T11:34:01Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"d38077d6-bda3-4a0b-8f1e-5e1b6d712a14\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Stopping instance\",\n\t\t\t\"created_at\": \"2022-12-15T11:34:01.127095932Z\",\n\t\t\t\"updated_at\": \"2022-12-15T11:34:01.127095932Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/v1\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:01Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/internal/containers/v1/onstopns?netns=%2Fproc%2F8311%2Ffd%2F4&project=default&target=stop" username=root
time="2022-12-15T11:34:01Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" http_code=200
time="2022-12-15T11:34:01Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/internal/containers/v1/onstop?project=default&target=stop" username=root
time="2022-12-15T11:34:01Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" http_code=200
time="2022-12-15T11:34:01Z" level=debug msg="Container stopped, cleaning up" instance=v1 instanceType=container project=default
time="2022-12-15T11:34:01Z" level=debug msg="Stopping device" device=root instance=v1 instanceType=container project=default type=disk
time="2022-12-15T11:34:01Z" level=debug msg="UnmountInstance started" instance=v1 project=default
time="2022-12-15T11:34:01Z" level=debug msg="Skipping unmount as in use" driver=lvm pool=vg_fast refCount=1 volName=v1
time="2022-12-15T11:34:01Z" level=debug msg="UnmountInstance finished" instance=v1 project=default
time="2022-12-15T11:34:01Z" level=debug msg="Instance operation lock finished" action=stop err="<nil>" instance=v1 project=default reusable=false
time="2022-12-15T11:34:01Z" level=debug msg="Scheduler: container v1 stopped: re-balancing"
time="2022-12-15T11:34:01Z" level=debug msg="Stop finished" instance=v1 instanceType=container project=default stateful=false
time="2022-12-15T11:34:01Z" level=debug msg="Success for operation" class=task description="Stopping instance" operation=d38077d6-bda3-4a0b-8f1e-5e1b6d712a14 project=default
time="2022-12-15T11:34:01Z" level=debug msg="Event listener server handler stopped" listener=24bfffb1-5c0a-4f72-9228-e79252bbd783 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:02Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=root
time="2022-12-15T11:34:02Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"192.168.2.12\",\n\t\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\",\n\t\t\t\t\"acme\",\n\t\t\t\t\"internal_metrics\",\n\t\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\t\"remote_token_expiry\",\n\t\t\t\t\"init_preseed\",\n\t\t\t\t\"storage_volumes_created_at\",\n\t\t\t\t\"cpu_hotplug\",\n\t\t\t\t\"projects_networks_zones\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"192.168.2.12:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw\\nHhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51\\neGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG\\nBSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP\\nfViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da\\np3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF\\nBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA\\nAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ\\nlAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO\\noaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"6.0.12-sm\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"2.9\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\": \"sm3\",\n\t\t\t\t\"server_pid\": 31569,\n\t\t\t\t\"server_version\": \"5.9\",\n\t\t\t\t\"storage\": \"lvm\",\n\t\t\t\t\"storage_version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"6.0.2\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:02Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/instances/v1 username=root
time="2022-12-15T11:34:02Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"architecture\": \"x86_64\",\n\t\t\t\"config\": {\n\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\"image.description\": \"Ubuntu jammy amd64 (20221214_07:42)\",\n\t\t\t\t\"image.os\": \"Ubuntu\",\n\t\t\t\t\"image.release\": \"jammy\",\n\t\t\t\t\"image.serial\": \"20221214_07:42\",\n\t\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\"volatile.base_image\": \"a1470837445b88444c0566b5a0c733ea86fc2fdc45d2eb4ad78c790e176ab9e9\",\n\t\t\t\t\"volatile.cloud-init.instance-id\": \"9fdebf18-8578-44b3-a57c-b6c4bf04ca34\",\n\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000}]\",\n\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000}]\",\n\t\t\t\t\"volatile.last_state.idmap\": \"[]\",\n\t\t\t\t\"volatile.last_state.power\": \"STOPPED\",\n\t\t\t\t\"volatile.last_state.ready\": \"false\",\n\t\t\t\t\"volatile.uuid\": \"4b982529-5b87-4e37-800c-e1a1a560a92c\"\n\t\t\t},\n\t\t\t\"devices\": {},\n\t\t\t\"ephemeral\": false,\n\t\t\t\"profiles\": [\n\t\t\t\t\"default\"\n\t\t\t],\n\t\t\t\"stateful\": false,\n\t\t\t\"description\": \"\",\n\t\t\t\"created_at\": \"2022-12-15T11:32:27.636648308Z\",\n\t\t\t\"expanded_config\": {\n\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\"image.description\": \"Ubuntu jammy amd64 (20221214_07:42)\",\n\t\t\t\t\"image.os\": \"Ubuntu\",\n\t\t\t\t\"image.release\": \"jammy\",\n\t\t\t\t\"image.serial\": \"20221214_07:42\",\n\t\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\"volatile.base_image\": \"a1470837445b88444c0566b5a0c733ea86fc2fdc45d2eb4ad78c790e176ab9e9\",\n\t\t\t\t\"volatile.cloud-init.instance-id\": \"9fdebf18-8578-44b3-a57c-b6c4bf04ca34\",\n\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000}]\",\n\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":1000000000}]\",\n\t\t\t\t\"volatile.last_state.idmap\": \"[]\",\n\t\t\t\t\"volatile.last_state.power\": \"STOPPED\",\n\t\t\t\t\"volatile.last_state.ready\": \"false\",\n\t\t\t\t\"volatile.uuid\": \"4b982529-5b87-4e37-800c-e1a1a560a92c\"\n\t\t\t},\n\t\t\t\"expanded_devices\": {\n\t\t\t\t\"root\": {\n\t\t\t\t\t\"path\": \"/\",\n\t\t\t\t\t\"pool\": \"vg_fast\",\n\t\t\t\t\t\"type\": \"disk\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"name\": \"v1\",\n\t\t\t\"status\": \"Stopped\",\n\t\t\t\"status_code\": 102,\n\t\t\t\"last_used_at\": \"2022-12-15T11:33:58.874331931Z\",\n\t\t\t\"location\": \"none\",\n\t\t\t\"type\": \"container\",\n\t\t\t\"project\": \"default\"\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:02Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=root
time="2022-12-15T11:34:02Z" level=debug msg="Event listener server handler started" id=b42c6f09-0acf-4306-829a-bc088691452d local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:02Z" level=debug msg="Handling API request" ip=@ method=PUT protocol=unix url=/1.0/instances/v1/state username=root
time="2022-12-15T11:34:02Z" level=debug msg="API Request\n\t{\n\t\t\"action\": \"start\",\n\t\t\"timeout\": 0,\n\t\t\"force\": false,\n\t\t\"stateful\": false\n\t}" ip=@ method=PUT protocol=unix url=/1.0/instances/v1/state username=root
time="2022-12-15T11:34:02Z" level=debug msg="New operation" class=task description="Starting instance" operation=c1e1ce62-9370-4b65-81f8-39046223efd9 project=default
time="2022-12-15T11:34:02Z" level=debug msg="Started operation" class=task description="Starting instance" operation=c1e1ce62-9370-4b65-81f8-39046223efd9 project=default
time="2022-12-15T11:34:02Z" level=debug msg="Start started" instance=v1 instanceType=container project=default stateful=false
time="2022-12-15T11:34:02Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/c1e1ce62-9370-4b65-81f8-39046223efd9\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"c1e1ce62-9370-4b65-81f8-39046223efd9\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Starting instance\",\n\t\t\t\"created_at\": \"2022-12-15T11:34:02.763081616Z\",\n\t\t\t\"updated_at\": \"2022-12-15T11:34:02.763081616Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/v1\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=202
time="2022-12-15T11:34:02Z" level=debug msg="Instance operation lock created" action=start instance=v1 project=default reusable=false
time="2022-12-15T11:34:02Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/operations/c1e1ce62-9370-4b65-81f8-39046223efd9 username=root
time="2022-12-15T11:34:02Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"c1e1ce62-9370-4b65-81f8-39046223efd9\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Starting instance\",\n\t\t\t\"created_at\": \"2022-12-15T11:34:02.763081616Z\",\n\t\t\t\"updated_at\": \"2022-12-15T11:34:02.763081616Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/v1\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:02Z" level=debug msg="MountInstance started" instance=v1 project=default
time="2022-12-15T11:34:02Z" level=debug msg="MountInstance finished" instance=v1 project=default
time="2022-12-15T11:34:02Z" level=debug msg="Starting device" device=root instance=v1 instanceType=container project=default type=disk
time="2022-12-15T11:34:02Z" level=info msg="Starting container" action=start created="2022-12-15 11:32:27.636648308 +0000 UTC" ephemeral=false instance=v1 instanceType=container project=default stateful=false used="2022-12-15 11:33:58.874331931 +0000 UTC"
time="2022-12-15T11:34:02Z" level=debug msg="UpdateInstanceBackupFile started" instance=v1 project=default
time="2022-12-15T11:34:02Z" level=debug msg="Skipping unmount as in use" driver=lvm pool=vg_fast refCount=2 volName=v1
time="2022-12-15T11:34:02Z" level=debug msg="UpdateInstanceBackupFile finished" instance=v1 project=default
time="2022-12-15T11:34:02Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/internal/containers/v1/onstart?project=default" username=root
time="2022-12-15T11:34:02Z" level=debug msg="Scheduler: container v1 started: re-balancing"
time="2022-12-15T11:34:02Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" http_code=200
time="2022-12-15T11:34:02Z" level=info msg="Started container" action=start created="2022-12-15 11:32:27.636648308 +0000 UTC" ephemeral=false instance=v1 instanceType=container project=default stateful=false used="2022-12-15 11:33:58.874331931 +0000 UTC"
time="2022-12-15T11:34:02Z" level=debug msg="Instance operation lock finished" action=start err="<nil>" instance=v1 project=default reusable=false
time="2022-12-15T11:34:02Z" level=debug msg="Start finished" instance=v1 instanceType=container project=default stateful=false
time="2022-12-15T11:34:02Z" level=debug msg="Success for operation" class=task description="Starting instance" operation=c1e1ce62-9370-4b65-81f8-39046223efd9 project=default
time="2022-12-15T11:34:02Z" level=debug msg="Event listener server handler stopped" listener=b42c6f09-0acf-4306-829a-bc088691452d local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:03Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=root
time="2022-12-15T11:34:03Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"192.168.2.12\",\n\t\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\",\n\t\t\t\t\"acme\",\n\t\t\t\t\"internal_metrics\",\n\t\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\t\"remote_token_expiry\",\n\t\t\t\t\"init_preseed\",\n\t\t\t\t\"storage_volumes_created_at\",\n\t\t\t\t\"cpu_hotplug\",\n\t\t\t\t\"projects_networks_zones\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"192.168.2.12:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw\\nHhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51\\neGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG\\nBSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP\\nfViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da\\np3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF\\nBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA\\nAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ\\nlAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO\\noaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"6.0.12-sm\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"2.9\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\": \"sm3\",\n\t\t\t\t\"server_pid\": 31569,\n\t\t\t\t\"server_version\": \"5.9\",\n\t\t\t\t\"storage\": \"lvm\",\n\t\t\t\t\"storage_version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"6.0.2\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:03Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=root
time="2022-12-15T11:34:03Z" level=debug msg="Event listener server handler started" id=8f837e86-af43-434a-b351-b67b1a7b9637 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:03Z" level=debug msg="Handling API request" ip=@ method=POST protocol=unix url=/1.0/instances/v1/exec username=root
time="2022-12-15T11:34:03Z" level=debug msg="API Request\n\t{\n\t\t\"command\": [\n\t\t\t\"echo\",\n\t\t\t\"alive\"\n\t\t],\n\t\t\"wait-for-websocket\": true,\n\t\t\"interactive\": true,\n\t\t\"environment\": {\n\t\t\t\"TERM\": \"tmux-256color\"\n\t\t},\n\t\t\"width\": 245,\n\t\t\"height\": 60,\n\t\t\"record-output\": false,\n\t\t\"user\": 0,\n\t\t\"group\": 0,\n\t\t\"cwd\": \"\"\n\t}" ip=@ method=POST protocol=unix url=/1.0/instances/v1/exec username=root
time="2022-12-15T11:34:08Z" level=debug msg="Error Response\n\t{\n\t\t\"type\": \"error\",\n\t\t\"status\": \"\",\n\t\t\"status_code\": 0,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 400,\n\t\t\"error\": \"Instance is not running\",\n\t\t\"metadata\": null\n\t}" http_code=400
time="2022-12-15T11:34:08Z" level=debug msg="Event listener server handler stopped" listener=8f837e86-af43-434a-b351-b67b1a7b9637 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:35Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=root
time="2022-12-15T11:34:35Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"192.168.2.12\",\n\t\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\",\n\t\t\t\t\"acme\",\n\t\t\t\t\"internal_metrics\",\n\t\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\t\"remote_token_expiry\",\n\t\t\t\t\"init_preseed\",\n\t\t\t\t\"storage_volumes_created_at\",\n\t\t\t\t\"cpu_hotplug\",\n\t\t\t\t\"projects_networks_zones\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"192.168.2.12:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw\\nHhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51\\neGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG\\nBSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP\\nfViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da\\np3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF\\nBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA\\nAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ\\nlAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO\\noaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"6.0.12-sm\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"2.9\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\": \"sm3\",\n\t\t\t\t\"server_pid\": 31569,\n\t\t\t\t\"server_version\": \"5.9\",\n\t\t\t\t\"storage\": \"lvm\",\n\t\t\t\t\"storage_version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"6.0.2\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
time="2022-12-15T11:34:35Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=root
time="2022-12-15T11:34:35Z" level=debug msg="Event listener server handler started" id=fbfa98e5-0327-4a2c-a4aa-7e2fe13081f2 local=/var/lib/lxd/unix.socket remote=@
time="2022-12-15T11:34:35Z" level=debug msg="Handling API request" ip=@ method=POST protocol=unix url=/1.0/instances/v1/exec username=root
time="2022-12-15T11:34:35Z" level=debug msg="API Request\n\t{\n\t\t\"command\": [\n\t\t\t\"echo\",\n\t\t\t\"alive\"\n\t\t],\n\t\t\"wait-for-websocket\": true,\n\t\t\"interactive\": true,\n\t\t\"environment\": {\n\t\t\t\"TERM\": \"tmux-256color\"\n\t\t},\n\t\t\"width\": 245,\n\t\t\"height\": 60,\n\t\t\"record-output\": false,\n\t\t\"user\": 0,\n\t\t\"group\": 0,\n\t\t\"cwd\": \"\"\n\t}" ip=@ method=POST protocol=unix url=/1.0/instances/v1/exec username=root
time="2022-12-15T11:34:40Z" level=debug msg="Error Response\n\t{\n\t\t\"type\": \"error\",\n\t\t\"status\": \"\",\n\t\t\"status_code\": 0,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 400,\n\t\t\"error\": \"Instance is not running\",\n\t\t\"metadata\": null\n\t}" http_code=400
time="2022-12-15T11:34:40Z" level=debug msg="Event listener server handler stopped" listener=fbfa98e5-0327-4a2c-a4aa-7e2fe13081f2 local=/var/lib/lxd/unix.socket remote=@
ewildgoose commented 1 year ago

OK, so I have now reproduced this with a container running ssh. While the container is apparently not responding to the lxc tooling, I can still login via ssh and to all appearances, the container is running normally and without trouble, network interfaces are present, etc

repro:

sm3 ~ # lxc start gitea --console
To detach from the console, press: <ctrl>+a q
 * Caching service dependencies ... [ ok ]
 * Mounting cgroup filesystem ... [ ok ]
mount: /sys/fs/cgroup/openrc: wrong fs type, bad option, bad superblock on openrc, missing codepage or helper program, or other error.
 * Remounting devtmpfs on /dev ... [ ok ]
 * Mounting /dev/shm ... [ ok ]
 * Create Static Devices Nodes in /dev ... [ ok ]
 * Creating user login records ... [ ok ]
 * Wiping /tmp directory ... [ ok ]
 * Bringing up network interface lo ...RTNETLINK answers: File exists
 [ ok ]
 * Updating /etc/mtab ... * Creating mtab symbolic link
 [ ok ]
 * Create Volatile Files and Directories ... [ ok ]
INIT: Entering runlevel: 3
 * Checking your configfile (/etc/syslog-ng/syslog-ng.conf) ... [ ok ]
 * Starting syslog-ng ... [ ok ]
 * Starting cronie ... [ ok ]
 * Starting gitea ... [ ok ]
 * Bringing up interface priv0
 *   Caching network module dependencies
 *   config_priv0 not specified; defaulting to DHCP
 *   dhcp ... *     Running udhcpc ...udhcpc: started, v1.34.1
udhcpc: broadcasting discover
udhcpc: broadcasting select for 192.168.242.63, server 192.168.242.1
udhcpc: lease of 192.168.242.63 obtained from 192.168.242.1, lease time 3600
 [ ok ]
 *     received address 192.168.242.63/24
 [ ok ]
 *   Waiting for tentative IPv6 addresses to complete DAD (5 seconds) ..
 [ ok ]
 * Starting sshd ... [ ok ]
 * Starting local ... [ ok ]

gitea login:
gitea login:
gitea login:

So after a few seconds, the console stops responding to keystrokes

I can then stop the console and ssh in

sm3 ~ # ssh root@git
gitea ~ # tail /var/log/messages -f
Dec 15 12:12:59 gitea sshd[861]: Server listening on 0.0.0.0 port 22.
Dec 15 12:12:59 gitea sshd[861]: Server listening on :: port 22.
Dec 15 12:13:00 gitea agetty[899]: /dev/console: cannot get controlling tty: Operation not permitted
Dec 15 12:13:00 gitea agetty[899]: /dev/console: cannot get controlling tty: Operation not permitted
Dec 15 12:13:00 gitea agetty[899]: /dev/console: cannot set process group: Inappropriate ioctl for device
Dec 15 12:13:27 gitea sshd[900]: Connection closed by authenticating user root 169.254.0.1 port 56844 [preauth]
Dec 15 12:13:47 gitea sshd[903]: Connection closed by authenticating user root 94.228.36.46 port 51715 [preauth]
Dec 15 12:14:05 gitea sshd[906]: Connection closed by authenticating user root 94.228.36.46 port 51717 [preauth]
Dec 15 12:14:09 gitea sshd[910]: Connection closed by authenticating user root 94.228.36.46 port 51718 [preauth]
Dec 15 12:16:45 gitea sshd[912]: Accepted publickey for root from 94.228.36.46 port 51739 ssh2: ED25519 SHA256:Cftt+GVOKWzrmYtuUB97vKo5Tn5o+InZ0RH6XVwH770

This instance also runs a webserver, and I can interact with that as expected. Basically I can't see any evidence that it's not functioning as expected, however, the lxc tooling is unable to interact with it, exec anything, or stop the instance.

Does this give some clues as to what could be wrong?

Note I think I can repro this at least 1 time in 10 starting an instance, so I can collect strace or better logging if I had some guidance? With a leg up on understanding what is happening behind the scenes in the tooling and a pointer to relevant parts of the code I can dig deeper?

Thanks

tomponline commented 1 year ago

Please can you provide the info from the github issue template too. Thanks

ewildgoose commented 1 year ago

I have now reproduced this on a different server running a similar age gentoo/openrc distribution (up to date as of around yesterday). This machine is a Ryzen 9 16 core machine on a server MB. Similar config kernel running 6.0.12

ewildgoose commented 1 year ago

Required information

sm3 ~ # lxc info
config:
  core.https_address: 192.168.2.12
  images.auto_update_interval: "0"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 192.168.2.12:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw
    HhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51
    eGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG
    BSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP
    fViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da
    p3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF
    BwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA
    AAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ
    lAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO
    oaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=
    -----END CERTIFICATE-----
  certificate_fingerprint: 144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd
  driver: lxc
  driver_version: 5.0.1
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.0.12-sm
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Gentoo
  os_version: "2.9"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: sm3
  server_pid: 31569
  server_version: "5.9"
  storage: lvm
  storage_version: 2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0
  storage_supported_drivers:
  - name: btrfs
    version: 6.0.2
    remote: false
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0
    remote: false
sm3 ~ # lxc stop gitea -f --timeout 5 --debug
DEBUG  [2022-12-15T12:32:27Z] Connecting to a local LXD over a Unix socket
DEBUG  [2022-12-15T12:32:27Z] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-12-15T12:32:27Z] Got response struct from LXD
DEBUG  [2022-12-15T12:32:27Z]
    {
        "config": {
            "core.https_address": "192.168.2.12",
            "images.auto_update_interval": "0"
        },
        "api_extensions": [
            "storage_zfs_remove_snapshots",
            "container_host_shutdown_timeout",
            "container_stop_priority",
            "container_syscall_filtering",
            "auth_pki",
            "container_last_used_at",
            "etag",
            "patch",
            "usb_devices",
            "https_allowed_credentials",
            "image_compression_algorithm",
            "directory_manipulation",
            "container_cpu_time",
            "storage_zfs_use_refquota",
            "storage_lvm_mount_options",
            "network",
            "profile_usedby",
            "container_push",
            "container_exec_recording",
            "certificate_update",
            "container_exec_signal_handling",
            "gpu_devices",
            "container_image_properties",
            "migration_progress",
            "id_map",
            "network_firewall_filtering",
            "network_routes",
            "storage",
            "file_delete",
            "file_append",
            "network_dhcp_expiry",
            "storage_lvm_vg_rename",
            "storage_lvm_thinpool_rename",
            "network_vlan",
            "image_create_aliases",
            "container_stateless_copy",
            "container_only_migration",
            "storage_zfs_clone_copy",
            "unix_device_rename",
            "storage_lvm_use_thinpool",
            "storage_rsync_bwlimit",
            "network_vxlan_interface",
            "storage_btrfs_mount_options",
            "entity_description",
            "image_force_refresh",
            "storage_lvm_lv_resizing",
            "id_map_base",
            "file_symlinks",
            "container_push_target",
            "network_vlan_physical",
            "storage_images_delete",
            "container_edit_metadata",
            "container_snapshot_stateful_migration",
            "storage_driver_ceph",
            "storage_ceph_user_name",
            "resource_limits",
            "storage_volatile_initial_source",
            "storage_ceph_force_osd_reuse",
            "storage_block_filesystem_btrfs",
            "resources",
            "kernel_limits",
            "storage_api_volume_rename",
            "macaroon_authentication",
            "network_sriov",
            "console",
            "restrict_devlxd",
            "migration_pre_copy",
            "infiniband",
            "maas_network",
            "devlxd_events",
            "proxy",
            "network_dhcp_gateway",
            "file_get_symlink",
            "network_leases",
            "unix_device_hotplug",
            "storage_api_local_volume_handling",
            "operation_description",
            "clustering",
            "event_lifecycle",
            "storage_api_remote_volume_handling",
            "nvidia_runtime",
            "container_mount_propagation",
            "container_backup",
            "devlxd_images",
            "container_local_cross_pool_handling",
            "proxy_unix",
            "proxy_udp",
            "clustering_join",
            "proxy_tcp_udp_multi_port_handling",
            "network_state",
            "proxy_unix_dac_properties",
            "container_protection_delete",
            "unix_priv_drop",
            "pprof_http",
            "proxy_haproxy_protocol",
            "network_hwaddr",
            "proxy_nat",
            "network_nat_order",
            "container_full",
            "candid_authentication",
            "backup_compression",
            "candid_config",
            "nvidia_runtime_config",
            "storage_api_volume_snapshots",
            "storage_unmapped",
            "projects",
            "candid_config_key",
            "network_vxlan_ttl",
            "container_incremental_copy",
            "usb_optional_vendorid",
            "snapshot_scheduling",
            "snapshot_schedule_aliases",
            "container_copy_project",
            "clustering_server_address",
            "clustering_image_replication",
            "container_protection_shift",
            "snapshot_expiry",
            "container_backup_override_pool",
            "snapshot_expiry_creation",
            "network_leases_location",
            "resources_cpu_socket",
            "resources_gpu",
            "resources_numa",
            "kernel_features",
            "id_map_current",
            "event_location",
            "storage_api_remote_volume_snapshots",
            "network_nat_address",
            "container_nic_routes",
            "rbac",
            "cluster_internal_copy",
            "seccomp_notify",
            "lxc_features",
            "container_nic_ipvlan",
            "network_vlan_sriov",
            "storage_cephfs",
            "container_nic_ipfilter",
            "resources_v2",
            "container_exec_user_group_cwd",
            "container_syscall_intercept",
            "container_disk_shift",
            "storage_shifted",
            "resources_infiniband",
            "daemon_storage",
            "instances",
            "image_types",
            "resources_disk_sata",
            "clustering_roles",
            "images_expiry",
            "resources_network_firmware",
            "backup_compression_algorithm",
            "ceph_data_pool_name",
            "container_syscall_intercept_mount",
            "compression_squashfs",
            "container_raw_mount",
            "container_nic_routed",
            "container_syscall_intercept_mount_fuse",
            "container_disk_ceph",
            "virtual-machines",
            "image_profiles",
            "clustering_architecture",
            "resources_disk_id",
            "storage_lvm_stripes",
            "vm_boot_priority",
            "unix_hotplug_devices",
            "api_filtering",
            "instance_nic_network",
            "clustering_sizing",
            "firewall_driver",
            "projects_limits",
            "container_syscall_intercept_hugetlbfs",
            "limits_hugepages",
            "container_nic_routed_gateway",
            "projects_restrictions",
            "custom_volume_snapshot_expiry",
            "volume_snapshot_scheduling",
            "trust_ca_certificates",
            "snapshot_disk_usage",
            "clustering_edit_roles",
            "container_nic_routed_host_address",
            "container_nic_ipvlan_gateway",
            "resources_usb_pci",
            "resources_cpu_threads_numa",
            "resources_cpu_core_die",
            "api_os",
            "container_nic_routed_host_table",
            "container_nic_ipvlan_host_table",
            "container_nic_ipvlan_mode",
            "resources_system",
            "images_push_relay",
            "network_dns_search",
            "container_nic_routed_limits",
            "instance_nic_bridged_vlan",
            "network_state_bond_bridge",
            "usedby_consistency",
            "custom_block_volumes",
            "clustering_failure_domains",
            "resources_gpu_mdev",
            "console_vga_type",
            "projects_limits_disk",
            "network_type_macvlan",
            "network_type_sriov",
            "container_syscall_intercept_bpf_devices",
            "network_type_ovn",
            "projects_networks",
            "projects_networks_restricted_uplinks",
            "custom_volume_backup",
            "backup_override_name",
            "storage_rsync_compression",
            "network_type_physical",
            "network_ovn_external_subnets",
            "network_ovn_nat",
            "network_ovn_external_routes_remove",
            "tpm_device_type",
            "storage_zfs_clone_copy_rebase",
            "gpu_mdev",
            "resources_pci_iommu",
            "resources_network_usb",
            "resources_disk_address",
            "network_physical_ovn_ingress_mode",
            "network_ovn_dhcp",
            "network_physical_routes_anycast",
            "projects_limits_instances",
            "network_state_vlan",
            "instance_nic_bridged_port_isolation",
            "instance_bulk_state_change",
            "network_gvrp",
            "instance_pool_move",
            "gpu_sriov",
            "pci_device_type",
            "storage_volume_state",
            "network_acl",
            "migration_stateful",
            "disk_state_quota",
            "storage_ceph_features",
            "projects_compression",
            "projects_images_remote_cache_expiry",
            "certificate_project",
            "network_ovn_acl",
            "projects_images_auto_update",
            "projects_restricted_cluster_target",
            "images_default_architecture",
            "network_ovn_acl_defaults",
            "gpu_mig",
            "project_usage",
            "network_bridge_acl",
            "warnings",
            "projects_restricted_backups_and_snapshots",
            "clustering_join_token",
            "clustering_description",
            "server_trusted_proxy",
            "clustering_update_cert",
            "storage_api_project",
            "server_instance_driver_operational",
            "server_supported_storage_drivers",
            "event_lifecycle_requestor_address",
            "resources_gpu_usb",
            "clustering_evacuation",
            "network_ovn_nat_address",
            "network_bgp",
            "network_forward",
            "custom_volume_refresh",
            "network_counters_errors_dropped",
            "metrics",
            "image_source_project",
            "clustering_config",
            "network_peer",
            "linux_sysctl",
            "network_dns",
            "ovn_nic_acceleration",
            "certificate_self_renewal",
            "instance_project_move",
            "storage_volume_project_move",
            "cloud_init",
            "network_dns_nat",
            "database_leader",
            "instance_all_projects",
            "clustering_groups",
            "ceph_rbd_du",
            "instance_get_full",
            "qemu_metrics",
            "gpu_mig_uuid",
            "event_project",
            "clustering_evacuation_live",
            "instance_allow_inconsistent_copy",
            "network_state_ovn",
            "storage_volume_api_filtering",
            "image_restrictions",
            "storage_zfs_export",
            "network_dns_records",
            "storage_zfs_reserve_space",
            "network_acl_log",
            "storage_zfs_blocksize",
            "metrics_cpu_seconds",
            "instance_snapshot_never",
            "certificate_token",
            "instance_nic_routed_neighbor_probe",
            "event_hub",
            "agent_nic_config",
            "projects_restricted_intercept",
            "metrics_authentication",
            "images_target_project",
            "cluster_migration_inconsistent_copy",
            "cluster_ovn_chassis",
            "container_syscall_intercept_sched_setscheduler",
            "storage_lvm_thinpool_metadata_size",
            "storage_volume_state_total",
            "instance_file_head",
            "instances_nic_host_name",
            "image_copy_profile",
            "container_syscall_intercept_sysinfo",
            "clustering_evacuation_mode",
            "resources_pci_vpd",
            "qemu_raw_conf",
            "storage_cephfs_fscache",
            "network_load_balancer",
            "vsock_api",
            "instance_ready_state",
            "network_bgp_holdtime",
            "storage_volumes_all_projects",
            "metrics_memory_oom_total",
            "storage_buckets",
            "storage_buckets_create_credentials",
            "metrics_cpu_effective_total",
            "projects_networks_restricted_access",
            "storage_buckets_local",
            "loki",
            "acme",
            "internal_metrics",
            "cluster_join_token_expiry",
            "remote_token_expiry",
            "init_preseed",
            "storage_volumes_created_at",
            "cpu_hotplug",
            "projects_networks_zones"
        ],
        "api_status": "stable",
        "api_version": "1.0",
        "auth": "trusted",
        "public": false,
        "auth_methods": [
            "tls"
        ],
        "environment": {
            "addresses": [
                "192.168.2.12:8443"
            ],
            "architectures": [
                "x86_64",
                "i686"
            ],
            "certificate": "-----BEGIN CERTIFICATE-----\nMIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw\nHhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51\neGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG\nBSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP\nfViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da\np3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF\nBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA\nAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ\nlAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO\noaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=\n-----END CERTIFICATE-----\n",
            "certificate_fingerprint": "144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd",
            "driver": "lxc",
            "driver_version": "5.0.1",
            "firewall": "xtables",
            "kernel": "Linux",
            "kernel_architecture": "x86_64",
            "kernel_features": {
                "idmapped_mounts": "true",
                "netnsid_getifaddrs": "true",
                "seccomp_listener": "true",
                "seccomp_listener_continue": "true",
                "shiftfs": "false",
                "uevent_injection": "true",
                "unpriv_fscaps": "true"
            },
            "kernel_version": "6.0.12-sm",
            "lxc_features": {
                "cgroup2": "true",
                "core_scheduling": "true",
                "devpts_fd": "true",
                "idmapped_mounts_v2": "true",
                "mount_injection_file": "true",
                "network_gateway_device_route": "true",
                "network_ipvlan": "true",
                "network_l2proxy": "true",
                "network_phys_macvlan_mtu": "true",
                "network_veth_router": "true",
                "pidfd": "true",
                "seccomp_allow_deny_syntax": "true",
                "seccomp_notify": "true",
                "seccomp_proxy_send_notify_fd": "true"
            },
            "os_name": "Gentoo",
            "os_version": "2.9",
            "project": "default",
            "server": "lxd",
            "server_clustered": false,
            "server_event_mode": "full-mesh",
            "server_name": "sm3",
            "server_pid": 31569,
            "server_version": "5.9",
            "storage": "lvm",
            "storage_version": "2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0",
            "storage_supported_drivers": [
                {
                    "Name": "btrfs",
                    "Version": "6.0.2",
                    "Remote": false
                },
                {
                    "Name": "dir",
                    "Version": "1",
                    "Remote": false
                },
                {
                    "Name": "lvm",
                    "Version": "2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20) / 4.47.0",
                    "Remote": false
                }
            ]
        }
    }
DEBUG  [2022-12-15T12:32:27Z] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG  [2022-12-15T12:32:27Z] Sending request to LXD                        etag= method=PUT url="http://unix.socket/1.0/instances/gitea/state"
DEBUG  [2022-12-15T12:32:27Z]
    {
        "action": "stop",
        "timeout": 5,
        "force": true,
        "stateful": false
    }
DEBUG  [2022-12-15T12:32:27Z] Got operation from LXD
DEBUG  [2022-12-15T12:32:27Z]
    {
        "id": "a016d227-953c-40e1-91b0-b9d0ab852266",
        "class": "task",
        "description": "Stopping instance",
        "created_at": "2022-12-15T12:32:27.729422392Z",
        "updated_at": "2022-12-15T12:32:27.729422392Z",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "instances": [
                "/1.0/instances/gitea"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "none"
    }
DEBUG  [2022-12-15T12:32:27Z] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/operations/a016d227-953c-40e1-91b0-b9d0ab852266"
DEBUG  [2022-12-15T12:32:27Z] Got response struct from LXD
DEBUG  [2022-12-15T12:32:27Z]
    {
        "id": "a016d227-953c-40e1-91b0-b9d0ab852266",
        "class": "task",
        "description": "Stopping instance",
        "created_at": "2022-12-15T12:32:27.729422392Z",
        "updated_at": "2022-12-15T12:32:27.729422392Z",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "instances": [
                "/1.0/instances/gitea"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "none"
    }

Error: The instance is already stopped
Try `lxc info --show-log gitea` for more info

However, at the time of the above:

sm3 ~ # ps -ef |grep 19076
root     19076     1  0 12:12 ?        00:00:00 [lxc monitor] /var/lib/lxd/containers gitea
1000000  19084 19076  0 12:12 pts/0    00:00:00 init [3]

And I can ssh in to the running container and interact with it, plus interact with the webservice it's running over the network

ewildgoose commented 1 year ago

For completeness, here is that other container which is currently "stuck"

sm3 ~ # lxc config show gitea --expanded
architecture: x86_64
config:
  volatile.base_image: 441cd18e4d1c972f1fcd07bc54e3895dd537632cacf0232792e05aa350edcd6e
  volatile.cloud-init.instance-id: 0c310106-9076-4638-9eba-31cda5788317
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.net0.host_name: veth9b9ceefa
  volatile.net0.hwaddr: 00:16:3e:fe:b2:d1
  volatile.priv0.host_name: vetha47c1922
  volatile.priv0.hwaddr: 00:16:3e:75:90:51
  volatile.uuid: 03ca99a5-0305-4d20-9c8c-8e533dadad03
devices:
  gitea-data:
    path: /var/lib/gitea
    pool: vg_fast
    source: gitea-data
    type: disk
  net0:
    ipv4.address: 91.220.24.154
    name: net0
    nictype: routed
    parent: br0
    type: nic
  portage-cache-binpkgs:
    path: /var/cache/binpkgs
    shift: "true"
    source: /mnt/portage/cache/binpkgs/guest_amd64
    type: disk
  portage-cache-distfiles:
    path: /var/cache/distfiles
    shift: "true"
    source: /mnt/portage/cache/distfiles
    type: disk
  portage-repos:
    path: /var/db/repos
    readonly: "true"
    shift: "true"
    source: /mnt/portage/repos
    type: disk
  priv0:
    name: priv0
    network: priv0
    type: nic
  root:
    path: /
    pool: vg_fast
    type: disk
ephemeral: false
profiles:
- default
- mount_portage_amd64
- inet_priv
stateful: false
description: ""

It has a bridged network, a private network, root filesystem and a large (20GB) mounted data directory for the instance volatile data

However, the symptoms seem to be equivalent to the other image demonstrated originally above. That default image doesn't have ssh enabled, nor any network services, so it's difficult for me to prove that it's actually still alive and running ok inside the container, but I suspect they all are. Just the interaction via the lxc tooling is freezing

tomponline commented 1 year ago

If you remove all of the type: disk devices does it still occur (except the root disk)?

ewildgoose commented 1 year ago

What do you mean remove? There is only the root "type disk" device in the v1 instance ?? What else can I remove?

Note, I have reproduced a similar situation further down this bug report on another instance "gitea", which is running gentoo/openrc, has extra network devices and has extra storage partitions. However, in the original problem report I'm not using this. That was done using vanilla images:ubuntu/jammy (so you can repro it easily on your side)

To answer your question more completely (please read info this in addition to a similar statement in the top comment). Yes, I believe I can repro this with containers from ubuntu with systemd and also containers with openrc (eg gentoo). It originally felt like it happened slightly more frequently if I used containers which had additional devices (eg networking devices) attached to them. However, I'm now not confident of that opinion.

I can also repro a similar (same?) symptom on two different physical machines. Both are running very similar software though, however, this should eliminate it being a hardware specific issue.

I now think that the frequency is more like 1 in 5 or less. It is definitely possible to have it lock up twice in a row, but equally I needed to restart it more than 10+ times recently to repro it (so fits Poisson distribution of a random variable quite nicely!)

Note for completeness: The reason I show details of the gitea instance in this report is because it happened to have some useful network facing services attached. So I can use it to demonstrate that although the lxc tooling cannot communicate with the instance, I CAN communicate with it via ssh, git and http. To all measurable tests it is running normally as far as I can tell, just that lxc cannot interact with it

Is this clearer?

tomponline commented 1 year ago

Right, I didn't see that one in all the posts. Makes sense now. (I'm only skip reading this at the moment as am working on other things, but wanted to try and rule out any external issues).

ewildgoose commented 1 year ago

Additional results:

I tried shutting down the gitea instance from within the instance. I was curious if the monitor process would be able to see it stopping (apparently not)

sm3 ~ # ssh root@git
gitea ~ # poweroff

Broadcast message from root@gitea (pts/1) (Thu Dec 15 12:53:57 2022):

The system is going down for system halt NOW!
gitea ~ # Connection to git closed by remote host.
Connection to git closed.
sm3 ~ # lxc list
^C
sm3 ~ # ps ax|grep mon
19076 ?        Ss     0:00 [lxc monitor] /var/lib/lxd/containers gitea
21028 pts/3    S+     0:00 grep --colour=auto mon
sm3 ~ # ps -ef |grep 19076
root     19076     1  0 12:12 ?        00:00:00 [lxc monitor] /var/lib/lxd/containers gitea
1000000  19084 19076  0 12:12 ?        00:00:00 [init] <defunct>
root     21036 19597  0 12:54 pts/3    00:00:00 grep --colour=auto 19076
sm3 ~ # strace -p 19076
strace: Process 19076 attached
read(28,

I think this narrows the scope of the area of the problem quite significantly. Can you point me towards some documentation on how the communication path happens between the lxc tooling and the container so that I can try to understand the specifics a bit better? Thanks

Edit: Not demonstrating evidence of this, but take it on faith for now. When I see a hang on shutdown of a container using the lxc tooling I see something similar to the above. So there is a running monitor process, but a "defunct" init process. So I don't understand the mechanism that is being used for communication, but it feels like there is some pipe here, which can get closed due to $something, leaving the monitor process reading/writing to a non functioning pipe?

ewildgoose commented 1 year ago

OK, so I can repro slightly different symptoms with another container.

Summary here is that if I don't run "lxc console dnscache", then the lxc tooling runs ok. I can enter the container with lxc exec, etc.

BUT. If I run "lxc console dnscache", then after a few seconds the console stops responding and at that point the lxc tooling all stops being able to contact the container. Curiously, if I leave the "lxc exec dnscache bash" connection open, then this continues to function, but obviously I can't restart it if I exit from it.

example: 

sm3 /var/db/repos/gentoo # lxc start dnscache
sm3 /var/db/repos/gentoo # tail /var/log/lxd/dnscache/console.log -f
 *     received address 192.168.242.34/24
 [ ok ]
 *   Waiting for tentative IPv6 addresses to complete DAD (5 seconds) .
 [ ok ]
 * Starting sshd ... [ ok ]
 * Starting service scan ... [ ok ]
 * Starting local ... [ ok ]

dnscache login: ^C
sm3 /var/db/repos/gentoo # lxc exec dnscache bash
dnscache ~ # echo "this is now after the lxc console has stopped responding"
this is now after the lxc console has stopped responding
dnscache ~ #

In another window:

sm3 ~ # lxc console dnscache
To detach from the console, press: <ctrl>+a q

dnscache login:
dnscache login:
dnscache login:
dnscache login:
...freezes here, after about 4-5 seconds

lxc monitor says:


location: none
metadata:
  context:
    ip: '@'
    method: GET
    protocol: unix
    url: /1.0
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-12-15T13:49:02.818061559Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    protocol: unix
    url: /1.0/events
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-12-15T13:49:02.819598949Z"
type: logging

location: none
metadata:
  context:
    id: ccfb402c-ccaf-42d2-a38a-28769d50346e
    local: /var/lib/lxd/unix.socket
    remote: '@'
  level: debug
  message: Event listener server handler started
timestamp: "2022-12-15T13:49:02.81982216Z"
type: logging

location: none
metadata:
  context:
    http_code: "200"
  level: debug
  message: "WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\":
    200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\":
    {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"192.168.2.12\",\n\t\t\t\t\"images.auto_update_interval\":
    \"0\"\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\",\n\t\t\t\t\"acme\",\n\t\t\t\t\"internal_metrics\",\n\t\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\t\"remote_token_expiry\",\n\t\t\t\t\"init_preseed\",\n\t\t\t\t\"storage_volumes_created_at\",\n\t\t\t\t\"cpu_hotplug\",\n\t\t\t\t\"projects_networks_zones\"\n\t\t\t],\n\t\t\t\"api_status\":
    \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\":
    false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\":
    {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"192.168.2.12:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\":
    [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\":
    \"-----BEGIN CERTIFICATE-----\\nMIIB/DCCAYGgAwIBAgIRAJ+zvFFnK3ftskxxu8MFnJ0wCgYIKoZIzj0EAwMwMTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEBzbTMw\\nHhcNMjIxMjEwMTgzNjI5WhcNMzIxMjA3MTgzNjI5WjAxMRwwGgYDVQQKExNsaW51\\neGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHNtMzB2MBAGByqGSM49AgEG\\nBSuBBAAiA2IABHFNpMoYLSrYte1dL/dhR3JGkXj5JdZHfs5ocpgvBmUmzWUReYNP\\nfViNoYtB9nOwAmxWVh6jjBRAVSUylqN4Mr/SrvwG+/J3KoL6VgOvMUQWxebqJ3da\\np3wBsEtlWrptHqNdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF\\nBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDc20zhwR/AAABhxAAAAAAAAAA\\nAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDSm/Xx2xT8yUGdaqYChx46fJIJ\\nlAn8RHdo3DIaUEclFiP200QtWAmUroEwps9Nj6ECMQDZIFyQMSBpaEcGBxkuIQIO\\noaluQsv0KXSlE37AiMNGJWm9ukHEjxvVJueE3tGRsuw=\\n-----END
    CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"144a30ed49d18d626106902f0a9b73788cb82911f65a6ee8f4f6a4145729a5cd\",\n\t\t\t\t\"driver\":
    \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\":
    \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\":
    {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\":
    \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\":
    \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\":
    \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\":
    \"6.0.12-sm\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\":
    \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\":
    \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\":
    \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\":
    \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\":
    \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\":
    \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\":
    \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\":
    \"2.9\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\":
    false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\":
    \"sm3\",\n\t\t\t\t\"server_pid\": 31569,\n\t\t\t\t\"server_version\": \"5.9\",\n\t\t\t\t\"storage\":
    \"lvm\",\n\t\t\t\t\"storage_version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20)
    / 4.47.0\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\":
    \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"6.0.2\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\":
    \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\":
    \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.14(2) (2021-10-20) / 1.02.181 (2021-10-20)
    / 4.47.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}"
timestamp: "2022-12-15T13:49:02.818901679Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: POST
    protocol: unix
    url: /1.0/instances/dnscache/console
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-12-15T13:49:02.820230553Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: POST
    protocol: unix
    url: /1.0/instances/dnscache/console
    username: root
  level: debug
  message: "API Request\n\t{\n\t\t\"width\": 244,\n\t\t\"height\": 79,\n\t\t\"type\":
    \"console\"\n\t}"
timestamp: "2022-12-15T13:49:02.82027485Z"
type: logging

location: none
metadata:
  context:
    class: websocket
    description: Showing console
    operation: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    project: default
  level: debug
  message: New operation
timestamp: "2022-12-15T13:49:02.831866539Z"
type: logging

location: none
metadata:
  context:
    class: websocket
    description: Showing console
    operation: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    project: default
  level: debug
  message: Started operation
timestamp: "2022-12-15T13:49:02.831942386Z"
type: logging

location: none
metadata:
  context:
    http_code: "202"
  level: debug
  message: "WriteJSON\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation
    created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac\",\n\t\t\"error_code\":
    0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac\",\n\t\t\t\"class\":
    \"websocket\",\n\t\t\t\"description\": \"Showing console\",\n\t\t\t\"created_at\":
    \"2022-12-15T13:49:02.822108976Z\",\n\t\t\t\"updated_at\": \"2022-12-15T13:49:02.822108976Z\",\n\t\t\t\"status\":
    \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\":
    [\n\t\t\t\t\t\"/1.0/containers/dnscache\"\n\t\t\t\t],\n\t\t\t\t\"instances\":
    [\n\t\t\t\t\t\"/1.0/instances/dnscache\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\":
    {\n\t\t\t\t\"fds\": {\n\t\t\t\t\t\"0\": \"84776cd1d97e43add33bf5506fc624d57c26d1f9192190e79a56427a710dbf5c\",\n\t\t\t\t\t\"control\":
    \"87ccaa0c65573cff4b9df634a4b500c2cd13534cd355e5f7868e86410a1ace78\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"may_cancel\":
    false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}"
timestamp: "2022-12-15T13:49:02.832059354Z"
type: logging

location: none
metadata:
  class: websocket
  created_at: "2022-12-15T13:49:02.822108976Z"
  description: Showing console
  err: ""
  id: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
  location: none
  may_cancel: false
  metadata:
    fds:
      "0": 84776cd1d97e43add33bf5506fc624d57c26d1f9192190e79a56427a710dbf5c
      control: 87ccaa0c65573cff4b9df634a4b500c2cd13534cd355e5f7868e86410a1ace78
  resources:
    containers:
    - /1.0/containers/dnscache
    instances:
    - /1.0/instances/dnscache
  status: Pending
  status_code: 105
  updated_at: "2022-12-15T13:49:02.822108976Z"
project: default
timestamp: "2022-12-15T13:49:02.831911403Z"
type: operation

location: none
metadata:
  class: websocket
  created_at: "2022-12-15T13:49:02.822108976Z"
  description: Showing console
  err: ""
  id: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
  location: none
  may_cancel: false
  metadata:
    fds:
      "0": 84776cd1d97e43add33bf5506fc624d57c26d1f9192190e79a56427a710dbf5c
      control: 87ccaa0c65573cff4b9df634a4b500c2cd13534cd355e5f7868e86410a1ace78
  resources:
    containers:
    - /1.0/containers/dnscache
    instances:
    - /1.0/instances/dnscache
  status: Running
  status_code: 103
  updated_at: "2022-12-15T13:49:02.822108976Z"
project: default
timestamp: "2022-12-15T13:49:02.831965344Z"
type: operation

location: none
metadata:
  context:
    class: websocket
    description: Showing console
    operation: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    project: default
  level: debug
  message: Connecting to operation
timestamp: "2022-12-15T13:49:02.832868891Z"
type: logging

location: none
metadata:
  context:
    class: websocket
    description: Showing console
    operation: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    project: default
  level: debug
  message: Connected to operation
timestamp: "2022-12-15T13:49:02.833003425Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    protocol: unix
    url: /1.0/operations/d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac/websocket?secret=87ccaa0c65573cff4b9df634a4b500c2cd13534cd355e5f7868e86410a1ace78
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-12-15T13:49:02.832806373Z"
type: logging

location: none
metadata:
  context:
    class: websocket
    description: Showing console
    operation: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    project: default
  level: debug
  message: Connected to operation
timestamp: "2022-12-15T13:49:02.833448735Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    protocol: unix
    url: /1.0/operations/d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac/websocket?secret=84776cd1d97e43add33bf5506fc624d57c26d1f9192190e79a56427a710dbf5c
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-12-15T13:49:02.833328485Z"
type: logging

location: none
metadata:
  context:
    class: websocket
    description: Showing console
    operation: d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    project: default
  level: debug
  message: Connecting to operation
timestamp: "2022-12-15T13:49:02.833370479Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    protocol: unix
    url: /1.0/operations/d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-12-15T13:49:02.833761411Z"
type: logging

location: none
metadata:
  context: {}
  level: debug
  message: Started mirroring websocket
timestamp: "2022-12-15T13:49:02.833892779Z"
type: logging

location: none
metadata:
  action: instance-console
  context:
    type: console
  source: /1.0/instances/dnscache
project: default
timestamp: "2022-12-15T13:49:02.833824327Z"
type: lifecycle

location: none
metadata:
  context:
    http_code: "200"
  level: debug
  message: "WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\":
    200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\":
    {\n\t\t\t\"id\": \"d0b0ed86-e14e-4e54-bff2-4e50a5c0e7ac\",\n\t\t\t\"class\": \"websocket\",\n\t\t\t\"description\":
    \"Showing console\",\n\t\t\t\"created_at\": \"2022-12-15T13:49:02.822108976Z\",\n\t\t\t\"updated_at\":
    \"2022-12-15T13:49:02.822108976Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\":
    103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/dnscache\"\n\t\t\t\t],\n\t\t\t\t\"instances\":
    [\n\t\t\t\t\t\"/1.0/instances/dnscache\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\":
    {\n\t\t\t\t\"fds\": {\n\t\t\t\t\t\"0\": \"84776cd1d97e43add33bf5506fc624d57c26d1f9192190e79a56427a710dbf5c\",\n\t\t\t\t\t\"control\":
    \"87ccaa0c65573cff4b9df634a4b500c2cd13534cd355e5f7868e86410a1ace78\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"may_cancel\":
    false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}"
timestamp: "2022-12-15T13:49:02.833878363Z"
type: logging

This lockup happens 100% of the time running console (other instances it's "occasional" and with those I think the lxc tooling lockup occurs regardless of whether I check the console or not)

This instance is running gentoo/openrc, it's config is very similar to the other instances (they all were created from a common template). Main features are a some extra net devices, and running the dnscache dns cache software.

sm3 ~ # lxc config show dnscache --expanded
architecture: x86_64
config:
  boot.autostart: "false"
  volatile.cloud-init.instance-id: d502dad6-6ff9-4475-90b9-bf3e1208aa4e
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.net0.host_name: veth5d60f55c
  volatile.net0.hwaddr: 00:16:3e:8d:05:33
  volatile.priv0.host_name: vethccf85dda
  volatile.priv0.hwaddr: 00:16:3e:78:31:b9
  volatile.uuid: c7304d49-c38d-4648-a7d2-b650060ee716
devices:
  net0:
    ipv4.address: 91.220.24.3
    name: net0
    nictype: routed
    parent: br0
    type: nic
  portage-cache-binpkgs:
    path: /var/cache/binpkgs
    shift: "true"
    source: /mnt/portage/cache/binpkgs/guest_amd64
    type: disk
  portage-cache-distfiles:
    path: /var/cache/distfiles
    shift: "true"
    source: /mnt/portage/cache/distfiles
    type: disk
  portage-repos:
    path: /var/db/repos
    readonly: "true"
    shift: "true"
    source: /mnt/portage/repos
    type: disk
  priv0:
    name: priv0
    network: priv0
    type: nic
  root:
    path: /
    pool: vg_fast
    type: disk
ephemeral: false
profiles:
- default
- mount_portage_amd64
- inet_priv
stateful: false
description: ""

Just to give a sense of the container from inside:

sm3 /var/db/repos/gentoo # lxc exec dnscache bash
dnscache ~ # pstree -p
init(1)-+-agetty(912)
        |-busybox(765)
        |-crond(444)
        |-sshd(850)
        |-svscan(878)-+-supervise(913)---dnscache(915)
        |             `-supervise(914)---multilog(916)
        `-syslog-ng(414)---syslog-ng(415)-+-{syslog-ng}(416)
                                          |-{syslog-ng}(417)
                                          |-{syslog-ng}(851)
                                          `-{syslog-ng}(852)

dnscache ~ # ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 91.220.24.3  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::216:3eff:fe8d:533  prefixlen 64  scopeid 0x20<link>
        ether 00:16:3e:8d:05:33  txqueuelen 1000  (Ethernet)
        RX packets 37  bytes 2540 (2.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39  bytes 4491 (4.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

priv0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.242.34  netmask 255.255.255.0  broadcast 192.168.242.255
        inet6 fd42:6980:7ca3:9f8a:216:3eff:fe78:31b9  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::216:3eff:fe78:31b9  prefixlen 64  scopeid 0x20<link>
        ether 00:16:3e:78:31:b9  txqueuelen 1000  (Ethernet)
        RX packets 14  bytes 1970 (1.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1790 (1.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Edit: The symptoms above remain identical if I remove the extra devices and profiles, ie if I boot with only profile "default" to get my rootfs only, then I see the same behaviour

ewildgoose commented 1 year ago

Should I create a separate issue that "lxc stop -f" will not shutdown this container?

(However, "kill -9 " will shutdown the process tree, so it's an issue of code control flow through the stop function, not a "kernel process is unkillable" issue)

tomponline commented 1 year ago

Should I create a separate issue that "lxc stop -f" will not shutdown this container?

(However, "kill -9 " will shutdown the process tree, so it's an issue of code control flow through the stop function, not a "kernel process is unkillable" issue)

No thank you. Its best to leave one issue (or two in this case, as you've already opened another for the related issue).

ewildgoose commented 1 year ago

Further investigation shows that kernel 5.14.12 doesn't exhibit these symptoms, caveat that I tested only for a few minutes, but the problem was usually reproducible within 4-5 seconds

However, I can reproduce the problem with kernel 6.0.12 and 6.1.4 kernel config is "mostly" copied forward from the 5.14.2 with "make oldconfig". lxc-checkconfig doesn't show anything missing

Can you confirm that kernel 6.x are tested and working with some kernel config (I think you generally develop with Ubuntu, correct?)?

Can you please shine some light on how the communication process happens between the host and the containers? I'm unclear how this process is happening and don't even know where to poke to debug this further? Perhaps even you can point me to points in the code I could add some debugging into to try and figure out what is happening?

Just to repeat the symptoms in case they are getting lost - "lxc console xx" is working as expected for some few seconds, and then it appears to lock up. Container continues to run OK afterwards. It's quite "schrodinger" in that the console only hangs after you start to watch it's output. Other lxc commands then start to block as well. This is reproducible across multiple machines, using different architectures and doesn't appear to be related to the guest container OS.

Thanks for some advice on debugging this

ewildgoose commented 1 year ago

Additional:

5.14.12 - works 5.15.1 - works 5.15.20 - works 5.15.40 - works 5.15.60 - works

5.15.70 - doesn't work 5.15.80 - doesn't work 5.15.81 - doesn't work 5.15.82 - doesn't work 5.15.83 - doesn't work 5.15.86 - doesn't work

(I've tried to use similar .config, said no to all new features)

Git bisect finds this change to be causing the breakage for me: f770fba096a6d https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.15.y&id=f770fba096a6d

After this commit I'm seeing some symptom similar to a loss of connectivity between the host tools and the guest. strace is showing a bunch of io_uring calls as I interact with the console, so this seems to fit that it's an io_uring change that might be causing the problem

Could one of the LXD devs please comment on further ideas here?

tomponline commented 1 year ago

@brauner any thoughts on how https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.15.y&id=f770fba096a6d may affect LXC and cause it to stop responding?

tomponline commented 1 year ago

After this commit I'm seeing some symptom similar to a loss of connectivity between the host tools and the guest. strace is showing a bunch of io_uring calls as I interact with the console, so this seems to fit that it's an io_uring change that might be causing the problem

It wouldn't surprise me, we've seen quite a few issues with io_uring, not just with containers, but with VMs as well. And the most common issue we've seen is that the call just hangs and prevents anything progressing.

tomponline commented 1 year ago

Opened https://github.com/lxc/lxc/issues/4256 for this.

mihalicyn commented 1 year ago

Additional:

5.14.12 - works 5.15.1 - works 5.15.20 - works 5.15.40 - works 5.15.60 - works

5.15.70 - doesn't work 5.15.80 - doesn't work 5.15.81 - doesn't work 5.15.82 - doesn't work 5.15.83 - doesn't work 5.15.86 - doesn't work

(I've tried to use similar .config, said no to all new features)

Git bisect finds this change to be causing the breakage for me: f770fba096a6d https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.15.y&id=f770fba096a6d

After this commit I'm seeing some symptom similar to a loss of connectivity between the host tools and the guest. strace is showing a bunch of io_uring calls as I interact with the console, so this seems to fit that it's an io_uring change that might be causing the problem

Could one of the LXD devs please comment on further ideas here?

@ewildgoose

Your bisecting is really helpful. Great job!

Couldn't you check if your kernel contains fixes for this commit (can be easily found in kernel git log by "io_uring: poll rework"): https://github.com/torvalds/linux/commit/c487a5ad48831afa6784b368ec40d0ee50f2fe1b https://github.com/torvalds/linux/commit/539bcb57da2f58886d7d5c17134236b0ec9cd15d https://github.com/torvalds/linux/commit/7fdbc5f014c3f71bc44673a2d6c5bb2d12d45f25 https://github.com/torvalds/linux/commit/e2c0cb7c0cc72939b61a7efee376206725796625 https://github.com/torvalds/linux/commit/9d2ad2947a53abf5e5e6527a9eeed50a3a4cbc72 https://github.com/torvalds/linux/commit/b98186aee22fa593bc8c6b2c5d839c2ee518bc8c https://github.com/torvalds/linux/commit/12ad3d2d6c5b0131a6052de91360849e3e154846 https://github.com/torvalds/linux/commit/a26a35e9019fd70bf3cf647dcfdae87abc7bacea https://github.com/torvalds/linux/commit/c84b8a3fef663933007e885535591b9d30bdc860 https://github.com/torvalds/linux/commit/61bc84c4008812d784c398cfb54118c1ba396dfc https://github.com/torvalds/linux/commit/2f3893437a4ebf2e892ca172e9e122841319d675

ewildgoose commented 1 year ago

Hi, I'm about to jump on a plane, but I can confirm problems persist with 6.0.12 and 6.1.4 and I think 6.0.15, could someone check if those commits were in those trees?

mihalicyn commented 1 year ago

@ewildgoose github shows for each commit the branches where it is presented, you just need to follow links above.

ewildgoose commented 1 year ago

Hi, I'm sorry, but I'm not sure what you want me to do here? The links above, all say that they were in 6.1-rcx, so that implies that they should make it to 6.1.1 release? I'm seeing this problem with 6.1.4, so that would imply that all these fixes are included?

I just tested 6.1.6, which includes another 4 fixes for io_uring: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=v6.1.6&qt=grep&q=io_uring

However, this kernel ALSO, stops responding to "lxc console" after a few seconds, when lxc is compiled using io_uring.

Note that I see more commits arriving in 6.2-rc, which seem potentially relevant (last 5 commits are new): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?h=v6.2-rc4&qt=grep&q=io_uring

So phrased another way, problem as reported above is continuing in 6.1.6, which is latest released kernel as I write this.

mihalicyn commented 1 year ago

Yep, all the commits above is present in 6.1. If you're able to reproduce the problem on 6.1 then possibly there are something else in io_uring. I've written that commits list as a reply to your original report where you've pointed 5.x kernel versions ;-)

ewildgoose commented 1 year ago

Hi, that comment is "in addition" to the comment prior. I mention the 6.x kernels there.

To keep clarity on the bottom comment of the thread

mihalicyn commented 1 year ago

Ugh. I'm sorry. My bad.