search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.39k stars 932 forks source link

Can't start containers / zpool didn't mount #13559

Closed killua-eu closed 5 months ago

killua-eu commented 6 months ago

Required information


# Issue description

This is a machine running for 3 years now, no change to setup done in ages. After a scheduled update, some of the machines didnt come up. The ones having the problem use zfs, which for some reason doesn't mount. This never happened before, so before doing something potentially silly, I'm here with this issue.

root@az01:/etc/zfs# zpool status zpool status pool: containers state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:13:52 with 0 errors on Sun May 12 00:37:53 2024 config:

NAME                                    STATE     READ WRITE CKSUM
containers                              ONLINE       0     0     0
  6f065d17-4fc8-2c46-8f88-5910e887569c  ONLINE       0     0     0

errors: No known data errors

root@az01:/etc/zfs# zfs get all | grep mounted containers mounted no - containers/buckets mounted no - containers/containers mounted no - containers/containers/az01-crawl2 mounted no - containers/containers/az01-gitea mounted no - containers/containers/az01-lamp mounted no - containers/containers/az01-mailtrain mounted no - containers/containers/az01-nodered mounted no - containers/containers/az01-proxy mounted no - containers/containers/az01-stackedit mounted no - containers/containers/az01-wekan mounted no - containers/custom mounted no - containers/deleted mounted no - containers/deleted/buckets mounted no - containers/deleted/containers mounted no - containers/deleted/custom mounted no - containers/deleted/images mounted no - containers/deleted/virtual-machines mounted no - containers/images mounted no - containers/virtual-machines mounted no -


# Information to attach

 - [ ] Any relevant kernel output (`dmesg`)
 - [*] Container log (`lxc info NAME --show-log`)

lxc info az01-lamp --show-log Name: az01-lamp Status: STOPPED Type: container Architecture: x86_64 Location: az01 Created: 2020/12/04 19:17 CET Last Used: 2023/10/04 15:20 CEST

Log:

 - [*] Container configuration (`lxc config show NAME --expanded`)

architecture: x86_64 config: image.architecture: amd64 image.description: Ubuntu bionic amd64 (20180813_07:42) image.os: Ubuntu image.release: bionic image.serial: "20180813_07:42" volatile.base_image: 1b94e1c0c88518cc1518138070ed2d4a680faed06cf10b94a8bf004f4017e3a3 volatile.eth0.hwaddr: 00:16:3e:04:bf:31 volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.power: RUNNING volatile.uuid: 8ca40132-18f1-4584-9143-589aaabda2c4 volatile.uuid.generation: 8ca40132-18f1-4584-9143-589aaabda2c4 devices: eth0: name: eth0 nictype: bridged parent: lxdfan0 type: nic ftp: bind: host connect: tcp:127.0.0.1:20-21 listen: tcp:10.0.1.11:20-21 type: proxy ftp-pasv: bind: host connect: tcp:127.0.0.1:13000-14000 listen: tcp:10.0.1.11:13000-14000 type: proxy ftps: bind: host connect: tcp:127.0.0.1:989-990 listen: tcp:10.0.1.11:989-990 type: proxy root: path: / pool: local type: disk ephemeral: false profiles:

$ lxc storage list
+------------+--------+-------------+---------+---------+
|    NAME    | DRIVER | DESCRIPTION | USED BY |  STATE  |
+------------+--------+-------------+---------+---------+
| containers | zfs    |             | 14      | CREATED |
+------------+--------+-------------+---------+---------+
| local      | dir    |             | 2       | CREATED |
+------------+--------+-------------+---------+---------+

$ lxc storage show containers
name: containers
description: ""
driver: zfs
status: Created
config: {}
used_by:
- /1.0/instances/az01-crawl2
- /1.0/instances/az01-crawl2/snapshots/snap0
- /1.0/instances/az01-gitea
- /1.0/instances/az01-gitea/snapshots/snap0
- /1.0/instances/az01-gitea/snapshots/snap1
- /1.0/instances/az01-lamp
- /1.0/instances/az01-mailtrain
- /1.0/instances/az01-mailtrain/snapshots/snap0
- /1.0/instances/az01-mailtrain/snapshots/snap1
- /1.0/instances/az01-nodered
- /1.0/instances/az01-nodered/snapshots/snap0
- /1.0/instances/az01-proxy
- /1.0/instances/az01-stackedit
- /1.0/instances/az01-wekan
locations:
- az01

$ lxc storage show local
name: local
description: ""
driver: dir
status: Created
config: {}
used_by:
- /1.0/instances/az01-crawl
- /1.0/profiles/default
locations:
- az01

$ lxc start az01-crawl

$ lxc start az01-lamp
Error: saving config file for the container failed
Try `lxc info --show-log az01-lamp` for more info

$ lxc list
+----------------+---------+---------------------+------+-----------+-----------+----------+
|      NAME      |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS | LOCATION |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-crawl     | RUNNING | 240.11.0.127 (eth0) |      | CONTAINER | 0         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-crawl2    | STOPPED |                     |      | CONTAINER | 1         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-gitea     | STOPPED |                     |      | CONTAINER | 2         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-lamp      | STOPPED |                     |      | CONTAINER | 0         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-mailtrain | STOPPED |                     |      | CONTAINER | 2         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-nodered   | STOPPED |                     |      | CONTAINER | 1         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-proxy     | STOPPED |                     |      | CONTAINER | 0         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-stackedit | STOPPED |                     |      | CONTAINER | 0         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
| az01-wekan     | STOPPED |                     |      | CONTAINER | 0         | az01     |
+----------------+---------+---------------------+------+-----------+-----------+----------+
simondeziel commented 5 months ago

This zpool issue suggest some major kernel version changes between the pool creation and its usage. zpool upgrade -v might tell you more about the feature mismatch.

tomponline commented 5 months ago

Please can support issues be opened in the support forum, thanks.

https://discourse.ubuntu.com/c/lxd/support/149