shiftfs on top of cifs is giving inconsistent results

[bitranox]

• Distribution: Ubuntu eoan
• Distribution version: Ubuntu eoan. Kernel 5.3.0-24

• The output of "lxc info" :

  config: {}
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  MIIFR...KbQw0g==
  -----END CERTIFICATE-----
  certificate_fingerprint: 54e9f...2fe8
  driver: lxc
  driver_version: 3.2.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  netnsid_getifaddrs: "true"
  seccomp_listener: "true"
  shiftfs: "true"
  uevent_injection: "true"
  unpriv_fscaps: "true"
  kernel_version: 5.3.0-24-generic
  lxc_features:
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  seccomp_notify: "true"
  project: default
  server: lxd
  server_clustered: false
  server_name: srv3.nondiscloseddomain.com
  server_pid: 1480
  server_version: "3.18"
  storage: dir
  storage_version: "1"

  • Kernel version: 5.3.0-24

  • LXC version: client and server 3.18

  • LXD version: 3.18

  • Storage backend in use: SSD, ext4

Issue description

copy fails to chown files on cifs mounted volumes, bind mounted into the container, on the normal container filesystem, no problem. The file gets actually created, but errors on chown'ing them !

without shiftfs, unprivileged : works, but seldom problems as mentioned below without shiftfs, privileged : works, now testing, cannot say if there are problems, they occur very seldom with shiftfs, privileged : works, but seldom problems as mentioned below with shiftfs, unprivileged : errors as decribed below

Steps to reproduce

on the host :

# fstab
cifs Mount Parameter : //192.168.168.1/softdev /media/srv-main-softdev cifs credentials=/etc/.smb-pwd,iocharset=utf8,file_mode=0777,dir_mode=0777,uid=1000,gid=10800,vers=3.0 0 0

# bind mount in the container , see container configuration below.

lxc exec vm-pydev /bin/bash
cd media/srv-main-softdev/test

ls -la
>total 0
>drwxrwxrwx 2 srvadmin ac-softdev 0 Dec 11 18:14 .
>drwxrwxrwx 2 srvadmin ac-softdev 0 Dec 11 17:28 ..

# touch is ok 
touch ./test.txt
ls -la
> drwxrwxrwx 2 srvadmin ac-softdev 0 Dec 11 18:18 .
> drwxrwxrwx 2 srvadmin ac-softdev 0 Dec 11 17:28 ..
> -rwxrwxrwx 1 srvadmin ac-softdev 0 Dec 11 18:18 test.txt

# but copy fails - it says File exists - in fact it errors when chown the file
cp ./test.txt ./test2.txt
> cp: cannot create regular file './test2.txt': File exists

# and yes - now it exists
ls -la
> drwxrwxrwx 2 srvadmin ac-softdev 0 Dec 11 18:19 .
> drwxrwxrwx 2 srvadmin ac-softdev 0 Dec 11 17:28 ..
> -rwxrwxrwx 1 srvadmin ac-softdev 0 Dec 11 18:18 test.txt
> -rwxrwxrwx 1 srvadmin ac-softdev 0 Dec 11 18:19 test2.txt

# tried that on different hosts on different containers - everywhere the same !

Information to attach

• [ ] Any relevant kernel output (dmesg) no errors on container and host

• [ ] Container log (lxc info NAME --show-log)

  lxc vm-pydev 20191211180031.769 WARN     cgfsng - cgroups/cgfsng.c:chowmod:1525 - No such file or directory - Failed to chown(/sys/fs/cgroup/unified//lxc.payload/vm-pydev/memory.oom.group, 1000000000, 0)

• [ ] Container configuration (lxc config show NAME --expanded)

  architecture: x86_64
  config:
  image.description: ubuntu 19.10 amd64
  image.release: artful
  image.version: "19.10"
  raw.idmap: both 1000 1000
  volatile.base_image: 2d253a...406ab4
  volatile.eth0.host_name: veth18a4d736
  volatile.eth0.hwaddr: 00:16:3e:7d:70:3b
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1001001,"Nsid":1001,"Maprange":999998999},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1001001,"Nsid":1001,"Maprange":999998999}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1001001,"Nsid":1001,"Maprange":999998999},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1001001,"Nsid":1001,"Maprange":999998999}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  devices:
  eth0:
  name: eth0
  nictype: bridged
  parent: my-bridge0
  type: nic
  root:
  path: /
  pool: default
  type: disk
  srv-main-softdev:
  path: /media/srv-main-softdev
  shift: "true"
  source: /media/srv-main-softdev
  type: disk
  ephemeral: false
  profiles:
  - vm-pydev
  stateful: false
  description: ""

• [ ] Main daemon log no errors

• [ ] Output of the client with --debug no errors

• [ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue) no errors

[stgraber]

This isn't a LXD bug as you're not actually interacting with anything LXD controlled here.

This is a kernel bug, quite possibly in shiftfs or in cifs or some interaction of the two.

@brauner should be able to investigate this and sort it out.

[bitranox]

@stgraber stefan, did You pass it to @brauner, or where should i post it ?

[stgraber]

I pinged @brauner, I expect him to respond here. Because we close an issue doesn't mean we don't keep looking at it or track it somewhere else.

I suspect @brauner will want you to file an Ubuntu kernel bug for this at https://launchpad.net/ubuntu/+source/linux/+filebug as that's where development of the shiftfs patchset happens currently.

[bitranox]

@stgraber thanks Stefane, so I wait for @brauner 's requests .... thanks a lot ! Robert, Vienna

[brauner]

I'm looking at this.