stgraber
commented
6 years ago The first part may be a kernel bug, effectively /sys being racy between bump of the number of VFs and them being registered in /sys.
The second part does sound like a LXD bug, we'll need to track down exactly what's going on there.
LXD will set the MAC address on the container's interface, that part is expected, but this should work and I'm unsure why it's showing eth0 there instead of eth1.
sw37th
Required information
Distribution: Ubuntu (server)
Distribution version: Ubuntu 18.04.1 LTS server
The output of "lxc info" or if that fails:
% lxc info config: {} api_extensions: - storage_zfs_remove_snapshots - container_host_shutdown_timeout - container_stop_priority - container_syscall_filtering - auth_pki - container_last_used_at - etag - patch - usb_devices - https_allowed_credentials - image_compression_algorithm - directory_manipulation - container_cpu_time - storage_zfs_use_refquota - storage_lvm_mount_options - network - profile_usedby - container_push - container_exec_recording - certificate_update - container_exec_signal_handling - gpu_devices - container_image_properties - migration_progress - id_map - network_firewall_filtering - network_routes - storage - file_delete - file_append - network_dhcp_expiry - storage_lvm_vg_rename - storage_lvm_thinpool_rename - network_vlan - image_create_aliases - container_stateless_copy - container_only_migration - storage_zfs_clone_copy - unix_device_rename - storage_lvm_use_thinpool - storage_rsync_bwlimit - network_vxlan_interface - storage_btrfs_mount_options - entity_description - image_force_refresh - storage_lvm_lv_resizing - id_map_base - file_symlinks - container_push_target - network_vlan_physical - storage_images_delete - container_edit_metadata - container_snapshot_stateful_migration - storage_driver_ceph - storage_ceph_user_name - resource_limits - storage_volatile_initial_source - storage_ceph_force_osd_reuse - storage_block_filesystem_btrfs - resources - kernel_limits - storage_api_volume_rename - macaroon_authentication - network_sriov - console - restrict_devlxd - migration_pre_copy - infiniband - maas_network - devlxd_events - proxy - network_dhcp_gateway - file_get_symlink - network_leases - unix_device_hotplug - storage_api_local_volume_handling - operation_description - clustering - event_lifecycle - storage_api_remote_volume_handling - nvidia_runtime api_status: stable api_version: "1.0" auth: trusted public: false auth_methods: - tls environment: environment: addresses: [] architectures: - x86_64 - i686 certificate: | -----BEGIN CERTIFICATE----- (snip) -----END CERTIFICATE----- certificate_fingerprint: (snip) driver: lxc driver_version: 3.0.1 kernel: Linux kernel_architecture: x86_64 kernel_version: 4.15.0-34-generic server: lxd server_pid: 1461 server_version: 3.0.1 storage: btrfs storage_version: 4.15.1 server_clustered: false server_name: hibikiSR-IOV enabled device: Intel X520-DA2 (82599ES)
Issue description
I tried to add SR-IOV device to running container by following command:
At the first time, it returns following error:
But the directory /sys/class/net/enp4s0f0/device/virtfn1/net has created by LXD at this time.
And after the second time, it returns:
It seems that LXD tries to rename host machine's SR-IOV virtual function device name
eth0to container's device nameeth1. But I'm not sure why LXD also tries to set the MAC address to virtual function.Steps to reproduce
% sudo ethtool enp4s0f0 Settings for enp4s0f0: Supported ports: [ FIBRE ] Supported link modes: 10000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 10000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: Unknown! Duplex: Unknown! (255) Port: Direct Attach Copper PHYAD: 0 Transceiver: internal Auto-negotiation: off Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: no
% lspci | grep 82599ES 04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
% lxc config show mycontainer --expanded architecture: x86_64 config: image.architecture: amd64 image.description: ubuntu 18.04 LTS amd64 (release) (20180911) image.label: release image.os: ubuntu image.release: bionic image.serial: "20180911" image.version: "18.04" volatile.base_image: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a volatile.eth0.hwaddr: 00:16:3e:3d:a4:a8 volatile.idmap.base: "0" volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.power: RUNNING devices: eth0: name: eth0 nictype: bridged parent: lxdbr0 type: nic root: path: / pool: default type: disk ephemeral: false profiles:
default stateful: false description: ""
nictype=sriovdevice as "eth1" to running container.(But the directory /sys/class/net/enp4s0f0/device/virtfn1/net has created by LXD at this time)
% ls -ld /sys/class/net/enp4s0f0/device/virtfn1/net drwxr-xr-x 3 root root 0 Sep 17 18:25 /sys/class/net/enp4s0f0/device/virtfn1/net % ls -l /sys/class/net/enp4s0f0/device/virtfn1/net total 0 drwxr-xr-x 5 root root 0 Sep 17 18:25 eth1
(Virtual functions are created)
% ip a (snip) 9: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 92:1a:2a:af:e1:73 brd ff:ff:ff:ff:ff:ff 10: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ee:b9:b0:57:a3:0e brd ff:ff:ff:ff:ff:ff (snip) 70: eth61: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ce:3b:f2:da:9c:cc brd ff:ff:ff:ff:ff:ff 71: eth62: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 16:4e:33:fa:19:2b brd ff:ff:ff:ff:ff:ff
(Now mycontainer has properties volatile.eth1.hwaddr and volatile.eth1.name)
% lxc config show mycontainer --expanded architecture: x86_64 config: image.architecture: amd64 image.description: ubuntu 18.04 LTS amd64 (release) (20180911) image.label: release image.os: ubuntu image.release: bionic image.serial: "20180911" image.version: "18.04" volatile.base_image: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a volatile.eth0.hwaddr: 00:16:3e:3d:a4:a8 volatile.eth1.hwaddr: 00:16:3e:fd:07:49 volatile.eth1.name: eth1 volatile.idmap.base: "0" volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.power: RUNNING devices: eth0: name: eth0 nictype: bridged parent: lxdbr0 type: nic root: path: / pool: default type: disk ephemeral: false profiles:
default stateful: false description: ""
Add
nictype=sriovdevice to container again.Stop container, and add sriov device, and start. It seems that LXD tries to rename network device "veth9VDXG7" to "eth0".
% lxc config device add mycontainer eth1 nic nictype=sriov parent=enp4s0f0 Device eth1 added to mycontainer
% lxc config show mycontainer --expanded architecture: x86_64 config: image.architecture: amd64 image.description: ubuntu 18.04 LTS amd64 (release) (20180911) image.label: release image.os: ubuntu image.release: bionic image.serial: "20180911" image.version: "18.04" volatile.base_image: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a volatile.eth0.hwaddr: 00:16:3e:3d:a4:a8 volatile.eth1.hwaddr: 00:16:3e:fd:07:49 volatile.eth1.name: eth1 volatile.idmap.base: "0" volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.power: STOPPED devices: eth0: name: eth0 nictype: bridged parent: lxdbr0 type: nic eth1: nictype: sriov parent: enp4s0f0 type: nic root: path: / pool: default type: disk ephemeral: false profiles:
% lxc start mycontainer Error: Failed to run: /usr/lib/lxd/lxd forkstart mycontainer /var/lib/lxd/containers /var/log/lxd/mycontainer/lxc.conf: Try
lxc info --show-log mycontainerfor more info% lxc info --show-log mycontainer Name: mycontainer Remote: unix:// Architecture: x86_64 Created: 2018/09/17 09:21 UTC Status: Stopped Type: persistent Profiles: default
Log:
lxc mycontainer 20180917093504.360 ERROR lxc_network - network.c:lxc_setup_netdev_in_child_namespaces:2875 - Failed to rename network device "veth9VDXG7" to "eth0": File exists lxc mycontainer 20180917093504.360 ERROR lxc_network - network.c:lxc_setup_network_in_child_namespaces:3034 - failed to setup netdev lxc mycontainer 20180917093504.360 ERROR lxc_conf - conf.c:lxc_setup:3389 - Failed to setup network lxc mycontainer 20180917093504.360 ERROR lxc_start - start.c:do_start:1219 - Failed to setup container "mycontainer" lxc mycontainer 20180917093504.360 ERROR lxc_sync - sync.c:sync_wait:57 - An error occurred in another process (expected sequence number 5) lxc mycontainer 20180917093504.428 WARN lxc_network - network.c:lxc_delete_network_priv:2556 - Failed to rename interface with index 9 from "eth1" to its initial name "eth0" lxc mycontainer 20180917093504.428 ERROR lxc_start - start.c:lxc_start:1887 - Failed to spawn container "mycontainer" lxc mycontainer 20180917093504.428 ERROR lxc_container - lxccontainer.c:wait_on_daemonized_start:834 - Received container state "ABORTING" instead of "RUNNING" lxc 20180917093504.438 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to receive response for command "get_state"
defaultto dummy profilenonicwhich it has no network device. It seems that failed to setup hw address for network device "eth1".% lxc profile list +---------+---------+ | NAME | USED BY | +---------+---------+ | default | 1 | +---------+---------+ | nonic | 0 | +---------+---------+ % lxc profile show nonic config: {} description: LXD profile without nic device devices: root: path: / pool: default type: disk name: nonic used_by: [] % lxc config edit mycontainer % lxc config show mycontainer --expanded architecture: x86_64 config: image.architecture: amd64 image.description: ubuntu 18.04 LTS amd64 (release) (20180911) image.label: release image.os: ubuntu image.release: bionic image.serial: "20180911" image.version: "18.04" volatile.base_image: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a volatile.eth1.hwaddr: 00:16:3e:fd:07:49 volatile.eth1.name: eth1 volatile.idmap.base: "0" volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.power: STOPPED devices: eth1: nictype: sriov parent: enp4s0f0 type: nic root: path: / pool: default type: disk ephemeral: false profiles: - nonic stateful: false description: "" % lxc start mycontainer Error: Failed to run: /usr/lib/lxd/lxd forkstart mycontainer /var/lib/lxd/containers /var/log/lxd/mycontainer/lxc.conf: Try `lxc info --show-log mycontainer` for more info % lxc info --show-log mycontainer Name: mycontainer Remote: unix:// Architecture: x86_64 Created: 2018/09/17 09:21 UTC Status: Stopped Type: persistent Profiles: nonic Log: lxc mycontainer 20180917094353.196 ERROR lxc_network - network.c:setup_hw_addr:2758 - Failed to perform ioctl: Operation not permitted lxc mycontainer 20180917094353.196 ERROR lxc_network - network.c:lxc_setup_netdev_in_child_namespaces:2899 - Failed to setup hw address for network device "eth1" lxc mycontainer 20180917094353.196 ERROR lxc_network - network.c:lxc_setup_network_in_child_namespaces:3034 - failed to setup netdev lxc mycontainer 20180917094353.196 ERROR lxc_conf - conf.c:lxc_setup:3389 - Failed to setup network lxc mycontainer 20180917094353.196 ERROR lxc_start - start.c:do_start:1219 - Failed to setup container "mycontainer" lxc mycontainer 20180917094353.196 ERROR lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5) lxc mycontainer 20180917094353.196 WARN lxc_network - network.c:lxc_delete_network_priv:2556 - Failed to rename interface with index 9 from "eth1" to its initial name "eth0" lxc mycontainer 20180917094353.196 ERROR lxc_container - lxccontainer.c:wait_on_daemonized_start:834 - Received container state "ABORTING" instead of "RUNNING" lxc mycontainer 20180917094353.196 ERROR lxc_start - start.c:__lxc_start:1887 - Failed to spawn container "mycontainer" lxc 20180917094353.206 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to receive response for command "get_state"Information to attach
[ ] Any relevant kernel output (
dmesg)[ ] Container log (
lxc info NAME --show-log) Pelase see 'Steps to reproduce'[ ] Container configuration (
lxc config show NAME --expanded) Pelase see 'Steps to reproduce'[ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
lvl=info msg="LXD 3.0.1 is starting in normal mode" path=/var/lib/lxd t=2018-09-17T18:21:06+0900 lvl=info msg="Kernel uid/gid map:" t=2018-09-17T18:21:06+0900 lvl=info msg=" - u 0 0 4294967295" t=2018-09-17T18:21:06+0900 lvl=info msg=" - g 0 0 4294967295" t=2018-09-17T18:21:06+0900 lvl=info msg="Configured LXD uid/gid map:" t=2018-09-17T18:21:06+0900 lvl=info msg=" - u 0 100000 65536" t=2018-09-17T18:21:06+0900 lvl=info msg=" - g 0 100000 65536" t=2018-09-17T18:21:06+0900 lvl=warn msg="AppArmor support has been disabled because of lack of kernel support" t=2018-09-17T18:21:06+0900 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-09-17T18:21:06+0900 lvl=info msg="Initializing local database" t=2018-09-17T18:21:06+0900 lvl=info msg="Initializing database gateway" t=2018-09-17T18:21:06+0900 address= id=1 lvl=info msg="Start database node" t=2018-09-17T18:21:06+0900 lvl=info msg="Raft: Restored from snapshot 1-507-1537175388113" t=2018-09-17T18:21:06+0900 lvl=info msg="Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]" t=2018-09-17T18:21:06+0900 lvl=info msg="Raft: Node at 0 [Leader] entering Leader state" t=2018-09-17T18:21:06+0900 lvl=info msg="LXD is socket activated" t=2018-09-17T18:21:06+0900 lvl=info msg="Starting /dev/lxd handler:" t=2018-09-17T18:21:06+0900 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock t=2018-09-17T18:21:06+0900 lvl=info msg="REST API daemon:" t=2018-09-17T18:21:06+0900 lvl=info msg=" - binding Unix socket" socket=/var/lib/lxd/unix.socket t=2018-09-17T18:21:06+0900 lvl=info msg="Initializing global database" t=2018-09-17T18:21:06+0900 lvl=info msg="Initializing storage pools" t=2018-09-17T18:21:07+0900 lvl=info msg="Initializing networks" t=2018-09-17T18:21:07+0900 lvl=info msg="Loading configuration" t=2018-09-17T18:21:08+0900 lvl=info msg="Pruning expired images" t=2018-09-17T18:21:08+0900 lvl=info msg="Done pruning expired images" t=2018-09-17T18:21:08+0900 lvl=info msg="Expiring log files" t=2018-09-17T18:21:08+0900 lvl=info msg="Updating instance types" t=2018-09-17T18:21:08+0900 lvl=info msg="Updating images" t=2018-09-17T18:21:08+0900 lvl=info msg="Done expiring log files" t=2018-09-17T18:21:08+0900 lvl=info msg="Done updating images" t=2018-09-17T18:21:13+0900 ephemeral=false lvl=info msg="Creating container" name=mycontainer t=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Created container" name=mycontainer t=2018-09-17T18:21:15+0900 lvl=info msg="Done updating instance types" t=2018-09-17T18:21:17+0900 action=start created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Starting container" name=mycontainer stateful=false t=2018-09-17T18:21:19+0900 used=1970-01-01T09:00:00+0900 action=start created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Started container" name=mycontainer stateful=false t=2018-09-17T18:21:19+0900 used=1970-01-01T09:00:00+0900 action=shutdown created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Shutting down container" name=mycontainer t=2018-09-17T18:33:40+0900 timeout=-1s used=2018-09-17T18:21:19+0900 action=shutdown created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Shut down container" name=mycontainer t=2018-09-17T18:33:41+0900 timeout=-1s used=2018-09-17T18:21:19+0900 action=start created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Starting container" name=mycontainer stateful=false t=2018-09-17T18:35:04+0900 used=2018-09-17T18:21:19+0900 action=start created=2018-09-17T18:21:15+0900 ephemeral=false lvl=eror msg="Failed starting container" name=mycontainer stateful=false t=2018-09-17T18:35:04+0900 used=2018-09-17T18:21:19+0900 action=stop created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Container initiated stop" name=mycontainer stateful=false t=2018-09-17T18:35:04+0900 used=2018-09-17T18:35:04+0900 action=start created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Starting container" name=mycontainer stateful=false t=2018-09-17T18:43:52+0900 used=2018-09-17T18:35:04+0900 action=start created=2018-09-17T18:21:15+0900 ephemeral=false lvl=eror msg="Failed starting container" name=mycontainer stateful=false t=2018-09-17T18:43:53+0900 used=2018-09-17T18:35:04+0900 action=stop created=2018-09-17T18:21:15+0900 ephemeral=false lvl=info msg="Container initiated stop" name=mycontainer stateful=false t=2018-09-17T18:43:53+0900 used=2018-09-17T18:43:52+0900[ ] Output of the client with --debug
% lxc start mycontainer --debug DBUG[09-17|18:57:22] Connecting to a local LXD over a Unix socket DBUG[09-17|18:57:22] Sending request to LXD etag= method=GET url=http://unix.socket/1.0 DBUG[09-17|18:57:22] Got response struct from LXD DBUG[09-17|18:57:22] { "config": {}, "api_extensions": [ "storage_zfs_remove_snapshots", "container_host_shutdown_timeout", "container_stop_priority", "container_syscall_filtering", "auth_pki", "container_last_used_at", "etag", "patch", "usb_devices", "https_allowed_credentials", "image_compression_algorithm", "directory_manipulation", "container_cpu_time", "storage_zfs_use_refquota", "storage_lvm_mount_options", "network", "profile_usedby", "container_push", "container_exec_recording", "certificate_update", "container_exec_signal_handling", "gpu_devices", "container_image_properties", "migration_progress", "id_map", "network_firewall_filtering", "network_routes", "storage", "file_delete", "file_append", "network_dhcp_expiry", "storage_lvm_vg_rename", "storage_lvm_thinpool_rename", "network_vlan", "image_create_aliases", "container_stateless_copy", "container_only_migration", "storage_zfs_clone_copy", "unix_device_rename", "storage_lvm_use_thinpool", "storage_rsync_bwlimit", "network_vxlan_interface", "storage_btrfs_mount_options", "entity_description", "image_force_refresh", "storage_lvm_lv_resizing", "id_map_base", "file_symlinks", "container_push_target", "network_vlan_physical", "storage_images_delete", "container_edit_metadata", "container_snapshot_stateful_migration", "storage_driver_ceph", "storage_ceph_user_name", "resource_limits", "storage_volatile_initial_source", "storage_ceph_force_osd_reuse", "storage_block_filesystem_btrfs", "resources", "kernel_limits", "storage_api_volume_rename", "macaroon_authentication", "network_sriov", "console", "restrict_devlxd", "migration_pre_copy", "infiniband", "maas_network", "devlxd_events", "proxy", "network_dhcp_gateway", "file_get_symlink", "network_leases", "unix_device_hotplug", "storage_api_local_volume_handling", "operation_description", "clustering", "event_lifecycle", "storage_api_remote_volume_handling", "nvidia_runtime" ], "api_status": "stable", "api_version": "1.0", "auth": "trusted", "public": false, "auth_methods": [ "tls" ], "environment": { "addresses": [], "architectures": [ "x86_64", "i686" ], "certificate": (snip)", "certificate_fingerprint": (snip)" "driver": "lxc", "driver_version": "3.0.1", "kernel": "Linux", "kernel_architecture": "x86_64", "kernel_version": "4.15.0-34-generic", "server": "lxd", "server_pid": 1408, "server_version": "3.0.1", "storage": "btrfs", "storage_version": "4.15.1", "server_clustered": false, "server_name": "hibiki" } } DBUG[09-17|18:57:22] Sending request to LXD etag= method=GET url=http://unix.socket/1.0/containers/mycontainer DBUG[09-17|18:57:22] Got response struct from LXD DBUG[09-17|18:57:22] { "architecture": "x86_64", "config": { "image.architecture": "amd64", "image.description": "ubuntu 18.04 LTS amd64 (release) (20180911)", "image.label": "release", "image.os": "ubuntu", "image.release": "bionic", "image.serial": "20180911", "image.version": "18.04", "volatile.base_image": "c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a", "volatile.eth1.hwaddr": "00:16:3e:fd:07:49", "volatile.eth1.name": "eth1", "volatile.idmap.base": "0", "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.last_state.power": "STOPPED" }, "devices": { "eth1": { "nictype": "sriov", "parent": "enp4s0f0", "type": "nic" } }, "ephemeral": false, "profiles": [ "nonic" ], "stateful": false, "description": "", "created_at": "2018-09-17T18:21:15+09:00", "expanded_config": { "image.architecture": "amd64", "image.description": "ubuntu 18.04 LTS amd64 (release) (20180911)", "image.label": "release", "image.os": "ubuntu", "image.release": "bionic", "image.serial": "20180911", "image.version": "18.04", "volatile.base_image": "c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a", "volatile.eth1.hwaddr": "00:16:3e:fd:07:49", "volatile.eth1.name": "eth1", "volatile.idmap.base": "0", "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.last_state.power": "STOPPED" }, "expanded_devices": { "eth1": { "nictype": "sriov", "parent": "enp4s0f0", "type": "nic" }, "root": { "path": "/", "pool": "default", "type": "disk" } }, "name": "mycontainer", "status": "Stopped", "status_code": 102, "last_used_at": "2018-09-17T18:43:52+09:00", "location": "" } DBUG[09-17|18:57:22] Connected to the websocket DBUG[09-17|18:57:22] Sending request to LXD etag= method=PUT url=http://unix.socket/1.0/containers/mycontainer/state DBUG[09-17|18:57:22] { "action": "start", "timeout": 0, "force": false, "stateful": false } DBUG[09-17|18:57:22] Got operation from LXD DBUG[09-17|18:57:22] { "id": "60a778d0-bac9-4520-ae45-19e5d2ff30a1", "class": "task", "description": "Starting container", "created_at": "2018-09-17T18:57:22.915196043+09:00", "updated_at": "2018-09-17T18:57:22.915196043+09:00", "status": "Running", "status_code": 103, "resources": { "containers": [ "/1.0/containers/mycontainer" ] }, "metadata": null, "may_cancel": false, "err": "" } DBUG[09-17|18:57:22] Sending request to LXD etag= method=GET url=http://unix.socket/1.0/operations/60a778d0-bac9-4520-ae45-19e5d2ff30a1 DBUG[09-17|18:57:22] Got response struct from LXD DBUG[09-17|18:57:22] { "id": "60a778d0-bac9-4520-ae45-19e5d2ff30a1", "class": "task", "description": "Starting container", "created_at": "2018-09-17T18:57:22.915196043+09:00", "updated_at": "2018-09-17T18:57:22.915196043+09:00", "status": "Running", "status_code": 103, "resources": { "containers": [ "/1.0/containers/mycontainer" ] }, "metadata": null, "may_cancel": false, "err": "" } Error: Failed to run: /usr/lib/lxd/lxd forkstart mycontainer /var/lib/lxd/containers /var/log/lxd/mycontainer/lxc.conf: Try `lxc info --show-log mycontainer` for more info[ ] Output of the daemon with --debug (alternatively output of
lxc monitorwhile reproducing the issue)% lxc monitor metadata: context: {} level: dbug message: 'New event listener: 34dc6751-465d-44f3-a58d-127e0559e833' timestamp: "2018-09-17T19:01:37.563202352+09:00" type: loggingmetadata: context: ip: '@' method: GET url: /1.0 level: dbug message: handling timestamp: "2018-09-17T19:01:43.33375673+09:00" type: logging
metadata: context: ip: '@' method: GET url: /1.0/containers/mycontainer level: dbug message: handling timestamp: "2018-09-17T19:01:43.336362077+09:00" type: logging
metadata: context: {} level: dbug message: 'New event listener: fd74f805-8347-4d8e-8be7-f599dc9a0add' timestamp: "2018-09-17T19:01:43.34282792+09:00" type: logging
metadata: context: ip: '@' method: GET url: /1.0/events level: dbug message: handling timestamp: "2018-09-17T19:01:43.342735668+09:00" type: logging
metadata: context: ip: '@' method: PUT url: /1.0/containers/mycontainer/state level: dbug message: handling timestamp: "2018-09-17T19:01:43.344078236+09:00" type: logging
metadata: context: {} level: dbug message: 'New task operation: 174d2384-463e-42c5-94e6-5040e1ec7d80' timestamp: "2018-09-17T19:01:43.398930855+09:00" type: logging
metadata: class: task created_at: "2018-09-17T19:01:43.350178884+09:00" description: Starting container err: "" id: 174d2384-463e-42c5-94e6-5040e1ec7d80 may_cancel: false metadata: null resources: containers:
metadata: context: {} level: dbug message: 'Started task operation: 174d2384-463e-42c5-94e6-5040e1ec7d80' timestamp: "2018-09-17T19:01:43.399000369+09:00" type: logging
metadata: class: task created_at: "2018-09-17T19:01:43.350178884+09:00" description: Starting container err: "" id: 174d2384-463e-42c5-94e6-5040e1ec7d80 may_cancel: false metadata: null resources: containers:
metadata: context: ip: '@' method: GET url: /1.0/operations/174d2384-463e-42c5-94e6-5040e1ec7d80 level: dbug message: handling timestamp: "2018-09-17T19:01:43.400456804+09:00" type: logging
metadata: context: {} level: dbug message: Initializing a BTRFS driver. timestamp: "2018-09-17T19:01:43.412834248+09:00" type: logging
metadata: action: container-updated source: /1.0/containers/mycontainer timestamp: "2018-09-17T19:01:43.461718781+09:00" type: lifecycle
metadata: context: {} level: dbug message: Mounting BTRFS storage volume for container "mycontainer" on storage pool "default". timestamp: "2018-09-17T19:01:43.462473985+09:00" type: logging
metadata: context: {} level: dbug message: Mounting BTRFS storage pool "default". timestamp: "2018-09-17T19:01:43.462505122+09:00" type: logging
metadata: context: {} level: dbug message: Mounted BTRFS storage volume for container "mycontainer" on storage pool "default". timestamp: "2018-09-17T19:01:43.462771857+09:00" type: logging
metadata: context: {} level: dbug message: Mounting BTRFS storage pool "default". timestamp: "2018-09-17T19:01:43.507641996+09:00" type: logging
metadata: context: {} level: dbug message: Mounting BTRFS storage volume for container "mycontainer" on storage pool "default". timestamp: "2018-09-17T19:01:43.507611928+09:00" type: logging
metadata: context: {} level: dbug message: Mounted BTRFS storage volume for container "mycontainer" on storage pool "default". timestamp: "2018-09-17T19:01:43.507993046+09:00" type: logging
metadata: context: action: start created: 2018-09-17 18:21:15 +0900 JST ephemeral: "false" name: mycontainer stateful: "false" used: 2018-09-17 18:57:23 +0900 JST level: info message: Starting container timestamp: "2018-09-17T19:01:43.508099952+09:00" type: logging
metadata: context: ip: '@' method: GET url: /1.0 level: dbug message: handling timestamp: "2018-09-17T19:01:43.530738919+09:00" type: logging
metadata: context: ip: '@' method: GET url: /internal/containers/6/onstart level: dbug message: handling timestamp: "2018-09-17T19:01:43.533392397+09:00" type: logging
metadata: context: {} level: dbug message: Initializing a BTRFS driver. timestamp: "2018-09-17T19:01:43.546040756+09:00" type: logging
metadata: context: {} level: dbug message: Mounting BTRFS storage pool "default". timestamp: "2018-09-17T19:01:43.546118774+09:00" type: logging
metadata: context: {} level: dbug message: Mounting BTRFS storage volume for container "mycontainer" on storage pool "default". timestamp: "2018-09-17T19:01:43.546090483+09:00" type: logging
metadata: context: {} level: dbug message: Mounted BTRFS storage volume for container "mycontainer" on storage pool "default". timestamp: "2018-09-17T19:01:43.546434099+09:00" type: logging
metadata: context: {} level: dbug message: 'Scheduler: container mycontainer started: re-balancing' timestamp: "2018-09-17T19:01:43.547732663+09:00" type: logging
metadata: context: {} level: dbug message: 'Failure for task operation: 174d2384-463e-42c5-94e6-5040e1ec7d80: Failed to run: /usr/lib/lxd/lxd forkstart mycontainer /var/lib/lxd/containers /var/log/lxd/mycontainer/lxc.conf: ' timestamp: "2018-09-17T19:01:43.727671396+09:00" type: logging
metadata: context: action: start created: 2018-09-17 18:21:15 +0900 JST ephemeral: "false" name: mycontainer stateful: "false" used: 2018-09-17 18:57:23 +0900 JST level: eror message: Failed starting container timestamp: "2018-09-17T19:01:43.727581966+09:00" type: logging
metadata: class: task created_at: "2018-09-17T19:01:43.350178884+09:00" description: Starting container err: 'Failed to run: /usr/lib/lxd/lxd forkstart mycontainer /var/lib/lxd/containers /var/log/lxd/mycontainer/lxc.conf: ' id: 174d2384-463e-42c5-94e6-5040e1ec7d80 may_cancel: false metadata: null resources: containers:
metadata: context: ip: '@' method: GET url: /1.0 level: dbug message: handling timestamp: "2018-09-17T19:01:44.246959005+09:00" type: logging
metadata: context: {} level: dbug message: 'Disconnected event listener: fd74f805-8347-4d8e-8be7-f599dc9a0add' timestamp: "2018-09-17T19:01:44.247094353+09:00" type: logging
metadata: context: ip: '@' method: GET url: /internal/containers/6/onstop?target=stop level: dbug message: handling timestamp: "2018-09-17T19:01:44.249238762+09:00" type: logging
metadata: context: {} level: dbug message: Initializing a BTRFS driver. timestamp: "2018-09-17T19:01:44.260199645+09:00" type: logging
metadata: context: action: stop created: 2018-09-17 18:21:15 +0900 JST ephemeral: "false" name: mycontainer stateful: "false" used: 2018-09-17 19:01:43 +0900 JST level: info message: Container initiated stop timestamp: "2018-09-17T19:01:44.2603251+09:00" type: logging
metadata: context: {} level: dbug message: 'Scheduler: container mycontainer stopped: re-balancing' timestamp: "2018-09-17T19:01:44.2608741+09:00" type: logging
metadata: context: {} level: dbug message: 'Scheduler: network: dev9 has been added: updating network priorities' timestamp: "2018-09-17T19:01:44.348180378+09:00" type: logging
^C