podman-in-podman: system reset: panic creating lock file directory

[edsantiago]

Seen just now in f38 containerized:

podman system reset
           [It] podman system reset
# podman [options] system reset -f
   A "/etc/containers/storage.conf" config file exists.
  Remove this file if you did not modify the configuration.
  panic: creating lock file directory: mkdir /tmp/podman_test1642220339/events: no such file or directory

  goroutine 9 [running]:
  github.com/containers/storage/pkg/lockfile.(*LockFile).lock(0xc00024b920, 0x1)
   /var/tmp/go/src/github.com/containers/podman/vendor/github.com[/containers/storage/pkg/lockfile/lockfile.go:397](https://github.com/containers/storage/blob/c9104cfc3350f0961418ff758878ba4859509985/pkg/lockfile/lockfile.go#L397) +0x145
  github.com/containers/storage/pkg/lockfile.(*LockFile).Lock(...)
   /var/tmp/go/src/github.com/containers/podman/vendor/github.com[/containers/storage/pkg/lockfile/lockfile.go:132](https://github.com/containers/storage/blob/c9104cfc3350f0961418ff758878ba4859509985/pkg/lockfile/lockfile.go#L132)
  github.com/containers/podman/v4/libpod/events.EventLogFile.Write({{{0x7fffb742afa2, 0x4}, {0xc0001ff710, 0x2c}, 0xf4240}}, {0x0, {0xc00004e840, 0x40}, {0x0, 0x0}, ...})
   /var/tmp/go/src/github.com[/containers/podman/libpod/events/logfile.go:51](https://github.com/containers/podman/blob/c9104cfc3350f0961418ff758878ba4859509985/libpod/events/logfile.go#L51) +0x119
  github.com/containers/podman/v4/libpod.(*Runtime).libimageEvents.func2()
   /var/tmp/go/src/github.com[/containers/podman/libpod/runtime.go:727](https://github.com/containers/podman/blob/c9104cfc3350f0961418ff758878ba4859509985/libpod/runtime.go#L727) +0x275
  created by github.com/containers/podman/v4/libpod.(*Runtime).libimageEvents
   /var/tmp/go/src/github.com[/containers/podman/libpod/runtime.go:713](https://github.com/containers/podman/blob/c9104cfc3350f0961418ff758878ba4859509985/libpod/runtime.go#L713) +0x10d

[mheon]

Theory: We're doing something with libimage after system reset deleted all directories associated with Podman, causing a panic as we can't create lock files in deleted directories.

[giuseppe]

could be a race condition where the go routine runs after the r.libimageRuntime.Shutdown() and we should wait for the goroutine to exit before .libimageRuntime.Shutdown() exits?

[mheon]

Events being a goroutine is unnecessary, usually, we can probably just convert it to a direct call to add an event

[edsantiago]

Could this be related to #17957?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

No further instances seen since this January 17 one. Searching my flake logs I see two others, both in March 2023, but those logs have long since evaporated. Copied from my logs:

• int podman fedora-37 root host sqlite

  [+1399s] Running: podman [...] rm -fa -t 0
  [+1399s] 62942d61068a7046a3ff6df59c6e16ce33107761e2643a24f5063ed8e37b995e
  [+1399s] output: 62942d61068a7046a3ff6df59c6e16ce33107761e2643a24f5063ed8e37b995e
  [+1399s] panic: creating lock file directory: mkdir /tmp/podman_test2754512739/root/vfs-containers: no such file or directory

  and

• int podman fedora-36 root host sqlite

  [+0940s] Verify podman containers.conf usage 
  [+0940s]   using journald for container with container log_tag
  [+0940s]   /var/tmp/go/src/github.com/containers/podman/test/e2e/containers_conf_test.go:230
  [no "Running" lines in log]
  [+0940s] panic: creating lock file directory: mkdir /tmp/podman_test2316220714/runroot/vfs-layers: no such file or directory

[edsantiago]

Not seen again since Jan 17.