unable to use 9pshare backed docker volumes for portainer data storage

Before you start please confirm the following.

[X] Yes, I've searched similar issues on GitHub.
[X] Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

We are using 9pshare backed docker volumes and portainer won't start.

Maybe this is related to virtio/9pshare not supporting every needed filesystem feature like locking?

Expected Behavior

We would like to use 9pshare backed docker volumes, which work in other places, with portainer as well.

Actual Behavior

The portainer container crashes/crashloops because it can't "open" the store.

We tested it with both existing files in the volume and an empty volume, same behaviour.

Steps to Reproduce

Our docker swarm is running on debian qemu virtual machines. The virtual machines use virtio/9pshare mounts to access a folder on the kvm host. The folder is a cephfs mount.

qemu config for 9pshare from the host side:

args: -fsdev local,security_model=passthrough,id=fsdev0,path=/mnt/cephfs/docker_volumes/production -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=9pshare,bus=pci.0,addr=0x4

in the debian vm we use this to consume the 9pshare mount and mount it in the same place:

mount -t 9p -o trans=virtio,version=9p2000.L,nobootwait,rw,_netdev,msize=262144 9pshare /mnt/cephfs/docker_volumes/production

now in our docker-compose/docker stack config, our volume just looks like this:

services:
  portainer:
    image: portainer/portainer-ce:2.19.4-alpine
    command: -H tcp://tasks.agent:9001 --tlsskipverify --log-level=DEBUG
    volumes:
      - /mnt/cephfs/docker_volumes/production/portainer/data:/data

We use docker volumes similar to this with other containers without problems (i.e. min.io).

Portainer logs or screenshots

2024/03/06 09:27AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2024/03/06 09:27AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2024/03/06 09:27AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2024/03/06 09:27AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid argument"

Portainer version

2.19.4

Portainer Edition

Community Edition (CE)

Platform and Version

Docker Swarm 25.0.3

OS and Architecture

debian 12

Browser

No response

What command did you use to deploy Portainer?

No response

Additional Information

No response

portainer / portainer