Unable to make instance snapshot: Error: Create instance snapshot: Invalid option for volume "<instance>/<snapshot>" option "volatile.uuid"

[tregubovav-dev]

Required information

• Distribution: Ubuntu
• Distribution version: 24.04
• The output of "incus info" or if that fails:
  • Kernel version: Linux picl-07 6.8.0-1010-raspi #11-Ubuntu SMP PREEMPT_DYNAMIC Thu Aug 8 23:22:41 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
  • LXC version: N/A
  • Incus version: 6.5
  • Storage backend in use: ceph (microceph 18.2.4+snapc9f2b08f92)

Issue description

This is a first time I tried to make snapshots after migrating from LXD to the Incus. And It's fail with multiple instances.

I'm not able to make instance snapshot. The operation fails with the message: Error: Create instance snapshot: Invalid option for volume "<instance>/<snapshot>" option "volatile.uuid" Note: All entities in enclosed in chevrons are obfuscated names. For example <instance> can be c1, inst-01, etc.

Steps to reproduce

1. Stop the instance: incus stop <instance> --project <project>
2. incus snapshot create --project Operstion fails with the message Error: Create instance snapshot: Invalid option for volume "<instance>/<snapshot>" option "volatile.uuid"

Information to attach

instance config

architecture: aarch64
config:
  boot.autostart: "true"
  cloud-init.user-data: |-
    #cloud-config
    runcmd:
    - <path_to_the_script>
  image.architecture: arm64
  image.description: Alpine edge arm64 (20231026_13:00)
  image.os: Alpine
  image.release: edge
  image.requirements.secureboot: "false"
  image.serial: "20231026_13:00"
  image.type: squashfs
  image.variant: cloud
  limits.cpu: "2"
  limits.memory: 256MB
  volatile.base_image: 137d60ec0178bc8ec11358f0a8bcb683ba4bb59c387f0040c357ca2168c8d303
  volatile.cloud-init.instance-id: e7b7fbfa-685e-4bc8-a46f-db85044f6822
  volatile.eth0.hwaddr: 00:16:3e:4c:5f:65
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 7c44639c-5d9b-40c0-a55e-258f070a85e8
  volatile.uuid.generation: 7c44639c-5d9b-40c0-a55e-258f070a85e8
devices:
  bootstrap:
    path: <path>
    pool: <cephfs_pool>
    readonly: "true"
    source: <cephfs_volume>
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: <ummanaged_bridge>
    type: nic
  root:
    path: /
    pool: <ceph_pool>
    type: disk
ephemeral: false
profiles:
- <profile>
stateful: false
description: ""

incus monitor output

DEBUG  [2024-09-10T11:19:45-07:00] Event listener server handler started         id=1a5a0cf8-629b-4038-941e-5fcf4aa4fbdb local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-09-10T11:19:49-07:00] Handling API request                          ip="<client_ip:port>" method=GET protocol=tls url=/1.0 username=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] Handling API request                          ip="<client_ip:port>" method=GET protocol=tls url="/1.0/events?project=infra" username=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2
DEBUG  [2024-09-10T11:19:49-07:00] Event listener server handler started         id=fc30fae6-8086-4d93-bd18-21b34b061b8f local="<server_ip:port>" remote="<client_ip:port>"
DEBUG  [2024-09-10T11:19:49-07:00] Handling API request                          ip="<client_ip:port>" method=POST protocol=tls url="/1.0/instances/<instance>/snapshots?project=infra" username=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] New operation                                 class=task description="Snapshotting instance" operation=16bfe110-655f-4eaf-ae52-f7e80c5c8f9c project=infra
DEBUG  [2024-09-10T11:19:49-07:00] Started operation                             class=task description="Snapshotting instance" operation=16bfe110-655f-4eaf-ae52-f7e80c5c8f9c project=infra
DEBUG  [2024-09-10T11:19:49-07:00] Instance operation lock created               action=create instance=<instance>/<snapshot> project=infra reusable=false
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] Matched trusted cert                          fingerprint=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2 subject="<subject_name>"
DEBUG  [2024-09-10T11:19:49-07:00] Handling API request                          ip="<client_ip:port>" method=GET protocol=tls url="/1.0/operations/16bfe110-655f-4eaf-ae52-f7e80c5c8f9c?project=infra" username=0ef92a191b841279c58d12111311352a99b0af14b5d8f01ffe16fba5e91089d2
INFO   [2024-09-10T11:19:49-07:00] Creating instance snapshot                    ephemeral=false instance=<instance>/<snapshot> instanceType=container project=infra
INFO   [2024-09-10T11:19:49-07:00] Created instance snapshot                     ephemeral=false instance=<instance>/<snapshot> instanceType=container project=infra
DEBUG  [2024-09-10T11:19:49-07:00] CreateInstanceSnapshot started                driver=ceph instance=<instance>/<snapshot> pool=remote project=infra src=<instance>
DEBUG  [2024-09-10T11:19:49-07:00] CreateInstanceSnapshot finished               driver=ceph instance=<instance>/<snapshot> pool=remote project=infra src=<instance>
DEBUG  [2024-09-10T11:19:49-07:00] Instance operation lock finished              action=create err="<nil>" instance=<instance>/<snapshot> project=infra reusable=false
DEBUG  [2024-09-10T11:19:49-07:00] Failure for operation                         class=task description="Snapshotting instance" err="Create instance snapshot: Invalid option for volume \"<instance>/<snapshot>\" option \"volatile.uuid\"" operation=16bfe110-655f-4eaf-ae52-f7e80c5c8f9c project=infra
DEBUG  [2024-09-10T11:19:49-07:00] Event listener server handler stopped         listener=fc30fae6-8086-4d93-bd18-21b34b061b8f local="<server_ip:port>" remote="client_ip:port"

[stgraber]

That's some LXD-specific config key that somehow made it past lxd-to-incus.

You should be able to fix that with a incus storage volume edit and removing the problematic volatile.uuid key from the volume.

(Closing as we do have logic to clear that in current lxd-to-incus, so this was most likely an issue for a while and only showed up now when making the snapshot)