Copying containers with snapshots fails on BTRFS

[NeonHorizon]

Required information

• Distribution: Ubuntu Server
• Distribution version: 18.04.3 LTS (fully patched)
• The output of "lxc info":

  config:
  core.https_address: '[::]:8443'
  core.trust_password: true
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses:
  architectures:
  - x86_64
  - i686
  certificate: |
  certificate_fingerprint:
  driver: lxc
  driver_version: 3.2.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  netnsid_getifaddrs: "false"
  seccomp_listener: "false"
  shiftfs: "false"
  uevent_injection: "false"
  unpriv_fscaps: "true"
  kernel_version: 4.15.0-66-generic
  lxc_features:
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  seccomp_notify: "true"
  project: default
  server: lxd
  server_clustered: false
  server_name: virtual4
  server_pid: 1141
  server_version: "3.18"
  storage: btrfs
  storage_version: "4.4"

  • Kernel version: 4.15.0-66-generic
  • LXC version: ?
  • LXD version: 3.18 (latest snap version)
  • Storage backend in use: BTRFS

Issue description

Copying a container between hosts fails if container has a snapshot

Steps to reproduce

lxc launch ubuntu:18.04 live:test-container lxc stop live:test-container lxc copy live:test-container backup:test-container

...works...

lxc delete backup:test-container lxc snapshot live:test-container test-snapshot lxc copy live:test-container backup:test-container

Error: Failed instance creation:

• https://##### Error transferring container data: exit status 1
• https://##### Error transferring container data: websocket: bad handshake

Information to attach

ssh backup cat /var/snap/lxd/common/lxd/logs/lxd.log | grep 'eror'

t=2019-11-09T19:50:09+0000 lvl=eror msg="Problem with btrfs receive: It seems that you have changed your default subvolume or you specify other subvolume to\nmount btrfs, try to remount this btrfs filesystem with fs tree, and run btrfs receive again!\n" t=2019-11-09T19:50:09+0000 lvl=eror msg="Error during migration sink" err="exit status 1" t=2019-11-09T19:50:10+0000 lvl=eror msg="Error during migration sink" err="websocket: bad handshake"

[stgraber]

Both source and target servers are running 3.18?

[NeonHorizon]

Yes, they are identical. We actually have 4 servers and the fault is reproducible between any pair.

We actually noticed when trying to copy a container back from a backup server (maintained with --refresh copies) but it seems even just a basic copy shows the issue.

[NeonHorizon]

Interestingly enough the updates which we are doing with copy --refresh are working fine.

[stgraber]

That part is normal, after the initial copy, any additional transfers done through refresh use rsync rather than btrfs send/receive.

[NeonHorizon]

Not sure if its helpful but deleting the snapshot makes it copyable again. lxc delete live:test-container/test-snapshot lxc copy live:test-container backup:test-container ....works....

[stgraber]

@monstermunchkin is currently rewriting the entire btrfs storage driver so this will be fixed as part of that. We just need to make sure we have a test in place which would catch such issues in the future.

[NeonHorizon]

Thanks for the info Stéphane, with ZFS support recently making its way into Ubuntu I was wondering if changing filesystems was the way to go for future deployments but it sounds like BTRFS is still getting active LXD support so I'll stick with it.

[stgraber]

Yeah, we're still actively looking after btrfs. For Ubuntu, we certainly have had better experiences with zfs than btrfs, but we have hundreds of thousand of users on btrfs too, especially on chromebooks, so we still very much care about it.

[stgraber]

I've confirmed that I can transfer a container with a bunch of snapshots over the network using current master (new btrfs implementation).