search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.32k stars 927 forks source link

Inconsistent bind mount behavior. #10914

Closed gene-vigelis closed 2 years ago

gene-vigelis commented 2 years ago

Required information

Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy

config: core.https_address: '[::]:8443' core.trust_password: true api_extensions:

Issue description

If I bind mount a directory under an already bind mounted directory in the config the container will not start with id_map errors. If I bind mount only the higher level directory in the config and start the container I can lxc device add ... the other lower level bind mounts (but the container must be running) and everything works as expected. I can define the extra mounts in a profile and add that profile only after the container starts as well.

Steps to reproduce

  1. Define a bind mount with a target such as /var/lib/docker with "shift: true"
  2. Define a bind mount with a target such as /var/lib/docker/volumes/postgres1 with "shift: true"
  3. Attempt to start container. This step will fail.
  4. Remove the second bind mount from step #2 in the configuration
  5. Start the container (it will start fine now)
  6. Run: lxc config device add container_name disk_label disk source=/xxx path=/var/lib/docker/volumes/postgre1 shift=true
  7. Log into the already running container and the bind mount that failed from step 2 is successfully added in step 6

Information to attach

stgraber commented 2 years ago

@gene-vigelis do you get the same behavior without shift=true?

brauner commented 2 years ago

Can you please get us the trace log for the container?

stgraber commented 2 years ago

lxc config set NAME raw.lxc=lxc.log.level=trace, then attempt to start the broken container and finally post lxc info NAME --show-log

gene-vigelis commented 2 years ago

@stgraber It's been a while since I tried without the shift but I believe it did start that way but my ownership's were all wrong.

Command history:

1663159109

lxc config set docker raw.lxc=lxc.log.level=trace

1663159216

lxc config show docker -e > show-container-e.out

1663159270

lxc start docker > lxc-start-err.out 2>&1

1663159332

lxc info --show-log docker > show-log-with-submounts-on-start.out 2>&1

1663159380

lxc profile remove docker docker-disks

1663159413

lxc start docker

1663159459

lxc profile add docker docker-disks

1663159503

lxc info --show-log docker > show-log-with-submounts-added-after-start.out 2>&1

Files: lxc-start-with-submounts.log show-container-e.log show-log-with-submounts-added-after-start.log show-log-with-submounts-on-start.log

brauner commented 2 years ago

So LXC is unhappy because:

lxc docker 20220914124111.394 ERROR    conf - ../src/src/lxc/conf.c:__lxc_idmapped_mounts_child:3050 - Invalid cross-device link - Failed to open target mountpoint 19/var/lib/docker/volumes/pgadmin1_config for detached idmapped mount 22:19/var/snap/lxd/common/lxd/devices/docker/disk.pgadmin1_config.var-lib-docker-volumes-pgadmin1_config
lxc docker 20220914124111.394 ERROR    conf - ../src/src/lxc/conf.c:lxc_idmapped_mounts_child:3096 - Invalid cross-device link - Failed to setup idmapped mount entries
lxc docker 20220914124111.394 ERROR    conf - ../src/src/lxc/conf.c:lxc_setup:4383 - Invalid cross-device link - Failed to attached detached idmapped mounts

It seems that /var/lib/docker/volumes/pgadmin1_config crosses a mount which LXC currently blocks when setting up idmapped mounts.

gene-vigelis commented 2 years ago

@brauner, I have lxd installed via snap. What do I need to do to get his patch?

stgraber commented 2 years ago

Needs to be merged into LXC and then backported to LXC 5.0 and then cherry-picked into the snap or wait for the next 5.0.x release of liblxc to have it be picked up automatically

gene-vigelis commented 2 years ago

I've install snap LXD so my lxc --version shows 5.5. How will i know that this has been back-ported? I have a workaround as shown above so would like to know when I can stop my workaround and have the container start properly with all disks.

Thank you so very much for such prompt response to my error!!!

gene-vigelis commented 1 year ago

Hi @stgraber,

My lxc driver is still at 5.0.1, is there any timeline for when liblxc will be patched/backported to lxd in the snap store?

Gene