High number of Image download tokens operations stuck in running state

[FrancescoDeSimone]

Required information

• Distribution: Ubuntu

• Distribution version: 22.04.4 LTS"

• The output of "snap list --all lxd core20 core22 core24 snapd":

  Name    Version         Rev    Tracking       Publisher   Notes
  core20  20240227        2264   latest/stable  canonical✓  base,disabled
  core20  20240416        2318   latest/stable  canonical✓  base
  core22  20240111        1122   latest/stable  canonical✓  base,disabled
  core22  20240408        1380   latest/stable  canonical✓  base
  lxd     5.21.1-10f4115  28322  latest/stable  canonical✓  in-cohort,held
  snapd   2.61.2          21184  latest/stable  canonical✓  snapd,held
  snapd   2.62            21465  latest/stable  canonical✓  snapd,disabled,held

• The output of "lxc info" or if that fails:

  • Kernel version:
  • LXC version:
  • LXD version:
  • Storage backend in use:

https://pastebin.ubuntu.com/p/WrGFywnTRr/plain/

Issue description

After deploying microcloud an high number of Image download token operation is create. lxc operation list I have 378 line with Image download token

output of lxc operation list: https://pastebin.canonical.com/p/8CMxXG4nXh/ Update 07/06/2024: I have not registered new operation on the first microcloud node since a week: https://pastebin.canonical.com/p/NGFnWn3Fz6/plain/

On the other 2 microcloud node now I have new operation regarding the first node microcloud node2: https://pastebin.canonical.com/p/DWtzVP7pYM/plain/ microcloud node3: https://pastebin.canonical.com/p/gbFW4vPtmk/plain/

Steps to reproduce

Information to attach

• [ ] Any relevant kernel output (dmesg)
• [ ] Container log (lxc info NAME --show-log)
• [ ] Container configuration (lxc config show NAME --expanded)
• [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• [ ] Output of the client with --debug
• [ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

[MusicDin]

Could be that there are many images that are being refreshed, but due to low download speeds it takes too long to complete, so new refreshes start.

How many images is there on the server? (lxc image ls -f csv | wc -l)

[FrancescoDeSimone]

hi @MusicDin, from all the projects there are 6 images