Failed creating instance from image: Failed to set apply_template volatile key: Failed to set volatile config: Failed to begin transaction: context deadline exceeded

[nobuto-m]

Required information

• Distribution: Ubuntu
• Distribution version: 22.04 LTS
• The output of "lxc info" or if that fails:
  • Kernel version: 5.15.0-78-generic #85-Ubuntu
  • LXD version: snap 5.0.2-838e1b2 24322 5.0/stable/… canonical✓ -
  • Storage backend in use: dir

Issue description

Derived from: https://bugs.launchpad.net/juju/+bug/2030280

Juju has an ability to provision LXD containers and by using that function, LXD container creations fails from time to time.

EDIT: Just to be clear, even if we have 3 LXD hosts managed by Juju, it's not a clustered LXD. LXD daemon is still acting as a single LXD host and Juju agent talks to the local LXD host

Steps to reproduce

I couldn't find an easy way to trigger the issue with LXD solely yet. But the following procedure should work.

1. Prepare a MAAS provider for Juju
2. Create 3 machines with Juju
3. Provision 8 LXD containers on top of those 3 machines respectively

$ juju add-machine -n 3

$ for m in {0..2}; do for _ in {0..7}; do juju add-machine lxd:$m done done

Information to attach

• [x] Any relevant kernel output (dmesg)
• [ ] Container log (lxc info NAME --show-log)
• [ ] Container configuration (lxc config show NAME --expanded)
• [x] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• [ ] Output of the client with --debug
• [x] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

[nobuto-m]

lxd.log

juju_machine-2.log

$ juju machines
Machine  State    Address          Inst id              Base          AZ       Message
0        started  192.168.151.102  helped-monkey        ubuntu@22.04  default  Deployed
0/lxd/0  started  192.168.151.124  juju-a3df1a-0-lxd-0  ubuntu@22.04  default  Container started
0/lxd/1  started  192.168.151.109  juju-a3df1a-0-lxd-1  ubuntu@22.04  default  Container started
0/lxd/2  started  192.168.151.107  juju-a3df1a-0-lxd-2  ubuntu@22.04  default  Container started
0/lxd/3  started  192.168.151.105  juju-a3df1a-0-lxd-3  ubuntu@22.04  default  Container started
0/lxd/4  started  192.168.151.106  juju-a3df1a-0-lxd-4  ubuntu@22.04  default  Container started
0/lxd/5  started  192.168.151.121  juju-a3df1a-0-lxd-5  ubuntu@22.04  default  Container started
0/lxd/6  started  192.168.151.123  juju-a3df1a-0-lxd-6  ubuntu@22.04  default  Container started
0/lxd/7  started  192.168.151.122  juju-a3df1a-0-lxd-7  ubuntu@22.04  default  Container started
1        started  192.168.151.103  alive-guppy          ubuntu@22.04  default  Deployed
1/lxd/0  started  192.168.151.127  juju-a3df1a-1-lxd-0  ubuntu@22.04  default  Container started
1/lxd/1  started  192.168.151.128  juju-a3df1a-1-lxd-1  ubuntu@22.04  default  Container started
1/lxd/2  started  192.168.151.129  juju-a3df1a-1-lxd-2  ubuntu@22.04  default  Container started
1/lxd/3  started  192.168.151.113  juju-a3df1a-1-lxd-3  ubuntu@22.04  default  Container started
1/lxd/4  started  192.168.151.115  juju-a3df1a-1-lxd-4  ubuntu@22.04  default  Container started
1/lxd/5  started  192.168.151.114  juju-a3df1a-1-lxd-5  ubuntu@22.04  default  Container started
1/lxd/6  started  192.168.151.116  juju-a3df1a-1-lxd-6  ubuntu@22.04  default  Container started
1/lxd/7  started  192.168.151.126  juju-a3df1a-1-lxd-7  ubuntu@22.04  default  Container started
2        started  192.168.151.104  huge-hyena           ubuntu@22.04  default  Deployed
2/lxd/0  down                      pending              ubuntu@22.04           Failed creating instance from image: Volume path "/var/snap/lxd/common/lxd/storage-pools/default/containers/juju-a3df...
2/lxd/1  down                      pending              ubuntu@22.04           Failed creating instance from image: Volume path "/var/snap/lxd/common/lxd/storage-pools/default/containers/juju-a3df...
2/lxd/2  started  192.168.151.112  juju-a3df1a-2-lxd-2  ubuntu@22.04  default  Container started
2/lxd/3  started  192.168.151.110  juju-a3df1a-2-lxd-3  ubuntu@22.04  default  Container started
2/lxd/4  started  192.168.151.125  juju-a3df1a-2-lxd-4  ubuntu@22.04  default  Container started
2/lxd/5  started  192.168.151.118  juju-a3df1a-2-lxd-5  ubuntu@22.04  default  Container started
2/lxd/6  started  192.168.151.108  juju-a3df1a-2-lxd-6  ubuntu@22.04  default  Container started
2/lxd/7  started  192.168.151.111  juju-a3df1a-2-lxd-7  ubuntu@22.04  default  Container started
2/lxd/8  started  192.168.151.117  juju-a3df1a-2-lxd-8  ubuntu@22.04  default  Container started

[nobuto-m]

juju-crashdump-ae404fd5-a480-470a-ba2a-a8cccefc261c.tar.gz

[nobuto-m]

Just to be clear, even if we have 3 LXD hosts managed by Juju, it's not a clustered LXD. LXD daemon is still acting as a single LXD host and Juju agent talks to the local LXD host

[tomponline]

@masnax do you think you could see if you could reproduce and investigate this issue?