`lxd recover` should better handle containers with `security.idmap.isolated=true`

[simondeziel]

I just went through a lxd recover after an accidental apt-get autopurge -y snapd (don't ask or maybe around a beer). This nuked LXD's DB but (fortunately) left the zpool intact for lxd recover to recover all the data.

On this server, there was this ganymede container configured with security.idmap.isolated=true and presumably volatile.idmap.base=1065536. This container has many volumes attached to it so idmap details need to be right otherwise those won't be remapped and will be inaccessible.

After a successful lxd recover, here's what the config of the last snapshot for that container looks like:

# lxc config show ganymede/snap224
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20220503_08:02)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20220503_08:02"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 25a8e58d35044329c803d0efedfe33c164595c7b9cdb82473c9cc1f57084a03d
  volatile.cloud-init.instance-id: 41adbd45-d290-4cbe-b9e8-bc2bd71321e5
  volatile.eth0.host_name: macf2dfda94
  volatile.eth0.hwaddr: 00:16:3e:ee:ca:d9
  volatile.eth0.last_state.created: "false"
  volatile.eth0.name: eth0
  volatile.idmap.base: "1131072"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1065536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1065536,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1131072,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1132082,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1065536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1065536,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 8e8962d4-6af0-4f16-ad3b-77166a904dc0
  volatile.uuid.generation: 8e8962d4-6af0-4f16-ad3b-77166a904dc0
created_at: 2024-05-09T05:28:43.271228289Z
expires_at: 0001-01-01T00:00:00Z
devices:
  data-simon:
    path: /data/simon
    source: /data/simon
    type: disk
  eth0:
    network: lxdbr0
    type: nic
ephemeral: false
expanded_config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20220503_08:02)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20220503_08:02"
  image.type: squashfs
  image.variant: default
  raw.idmap: both 1000-1009 1000-1009
  security.devlxd: "false"
  security.idmap.isolated: "true"
  security.nesting: "true"
  security.privileged: "false"
  security.protection.delete: "true"
  security.syscalls.deny: |-
    io_uring_setup
    io_uring_enter
    io_uring_register
  security.syscalls.deny_compat: "true"
  snapshots.expiry: 3d
  snapshots.schedule: '@daily, @startup'
  volatile.base_image: 25a8e58d35044329c803d0efedfe33c164595c7b9cdb82473c9cc1f57084a03d
  volatile.cloud-init.instance-id: 41adbd45-d290-4cbe-b9e8-bc2bd71321e5
  volatile.eth0.host_name: macf2dfda94
  volatile.eth0.hwaddr: 00:16:3e:ee:ca:d9
  volatile.eth0.last_state.created: "false"
  volatile.eth0.name: eth0
  volatile.idmap.base: "1131072"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1065536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1065536,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1131072,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1132082,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1065536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1065536,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 8e8962d4-6af0-4f16-ad3b-77166a904dc0
  volatile.uuid.generation: 8e8962d4-6af0-4f16-ad3b-77166a904dc0
expanded_devices:
  data-simon:
    path: /data/simon
    source: /data/simon
    type: disk
  eth0:
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    size: 4GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap224
profiles:
- sdeziel
stateful: false
size: 700416

In there, we see that both volatile.idmap.current and volatile.last_state.idmap use a hostid of 1065536. We also see that volatile.idmap.next uses a different hostid of 1131072, presumably due to volatile.idmap.base being set to this value.

This will cause the container to go through an ID remapping which could have been avoided had volatile.idmap.next been set identically to volatile.idmap.current.

In otherwords, why is the volatile.idmap.base changed during recovery?

Additional information:

# snap list lxd
Name  Version         Rev    Tracking     Publisher   Notes
lxd   5.21.1-d46c406  28460  5.21/stable  canonical✓  -

[simondeziel]

For the record, I worked around this issue with this hack:

for ctn in apt ... log mail metrics pm rproxy smb squid weechat; do
    last_snap="$(lxc info "${ctn}" | grep -owE 'snap[0-9]+' | tail -n1)"
    lxc restore "${ctn}" "${last_snap}"
    last_state_idmap="$(lxc config get "${ctn}" volatile.last_state.idmap)"
    last_idmap_base="$(echo "${last_state_idmap}" | sed 's/.*"Hostid":\([0-9]\+\),.*/\1/')"
    lxc config set "${ctn}" volatile.idmap.base "${last_state_base}"
    lxc config set "${ctn}" volatile.idmap.next "${last_state_idmap}"
    lxc start "${ctn}"
done