lxc publish expands sparse files

[jlec]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04.3
• The output of "lxc info" or if that fails:

  $ lxc info
  config:
  core.https_address: 10.96.240.101:8443
  core.proxy_http: http://proxy.srv.scpdev:3128
  core.proxy_https: http://proxy.srv.scpdev:3128
  core.proxy_ignore_hosts: localhost,127.0.0.1,127.0.0.2,.srv.scpdev,.prim.scpdev,.mgt.scpdev,.astrazeneca.net
  core.trust_password: true
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses:
  - 10.96.240.101:8443
  architectures:
  - x86_64
  - i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
  certificate_fingerprint: 3712265d670b67020c93a4480faf539c4273b8a172ed5a0e7d34372a9cc27e47
  driver: lxc
  driver_version: 2.0.8
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-101-generic
  server: lxd
  server_pid: 6208
  server_version: "2.21"
  storage: zfs
  storage_version: 0.6.5.6-0ubuntu16

  Issue description

When publishing a snapshot of a normal sized container which holds a sparse file, the sparse file gets expanded. This creates massive images.

In my image I have the following situation

[root@ipa01 log]# du -h /var/log/lastlog
569K    /var/log/lastlog
[root@ipa01 log]# du -h --apparent-size lastlog
429G    lastlog
[root@ipa01 log]# ls -lsh lastlog
569K -rw-r--r-- 1 root root 429G Feb 10 12:24 lastlog

and the resulting image grows to nearly half a TB.

Steps to reproduce

1. Launch container and create a large sparse file
2. lxc snapshot CONTAINER backup
3. lxc publish CONTAINER/backup and watch /var/lib/lxd/images grow

Information to attach

• [x] Container log (lxc info NAME --show-log)

  
  Name: ipa-dns00
  Remote: unix://
  Architecture: x86_64
  Created: 2017/11/06 05:01 UTC
  Status: Running
  Type: persistent
  Profiles: M, scp, net_prim, net_ext
  Pid: 14606
  Ips:
  brext:    inet    10.96.224.8 vethO1386O
  brext:    inet6   fe80::216:3eff:fe1e:c499    vethO1386O
  brprim:   inet    10.96.241.12
  brprim:   inet6   fe80::216:3eff:fedf:b7bc
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
  Resources:
  Processes: 423
  Disk usage:
  root: 2.59GB
  CPU usage:
  CPU usage (in seconds): 1019
  Memory usage:
  Memory (current): 1.49GB
  Memory (peak): 1.58GB
  Network usage:
  brprim:
    Bytes received: 85.42MB
    Bytes sent: 815.61kB
    Packets received: 577534
    Packets sent: 8570
  lo:
    Bytes received: 5.82MB
    Bytes sent: 5.82MB
    Packets received: 15837
    Packets sent: 15837
  brext:
    Bytes received: 46.42MB
    Bytes sent: 30.63MB
    Packets received: 250355
    Packets sent: 139335
  Snapshots:
  20180210 (taken at 2018/02/10 12:03 UTC) (stateless)

Log:

        lxc 20180208214634.358 ERROR    lxc_utils - utils.c:safe_mount:1739 - Too many levels of symbolic links - Failed to mount /proc/sys/fs/binfmt_misc onto /usr/lib/x86_64-linux-gnu/lxc/proc/sys/fs/binfmt_misc

 - [x] Container configuration (`lxc config show NAME --expanded`)

architecture: x86_64 config: image.description: SCP Centos 7.3 (2017-09-04) limits.cpu: "8" limits.memory: 8GB raw.lxc: lxc.aa_profile=unconfined security.privileged: "true" user.network-config: | version: 1 config:

• type: physical name: brprim subnets:
  • type: dhcp gateway: "10.96.240.1" dns_servers: [10.96.240.7, 10.96.240.8, 10.96.240.9] user.user-data: |

    cloud-config

    bootcmd:

• mkdir /var/lib/home users:
• name: scp uid: "900" homedir: "/var/lib/home/scp" shell: /bin/bash sudo: ALL=(ALL) NOPASSWD:ALL volatile.base_image: 480eb9e52129349cf19251992b644147b4c046d36f21ad6926b39923f46e6699 volatile.brext.hwaddr: 00:16:3e:1e:c4:99 volatile.brprim.hwaddr: 00:16:3e:df:b7:bc volatile.idmap.base: "0" volatile.idmap.next: '[]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING devices: backup: path: /mnt/backup source: /mnt/backup type: disk brext: name: brext nictype: bridged parent: brext type: nic brprim: name: brprim nictype: bridged parent: brprim type: nic data: path: /mnt/data source: /mnt/data type: disk root: path: / pool: default type: disk shared: path: /mnt/shared source: /mnt/shared type: disk ephemeral: false profiles:
  • M
  • scp
  • net_prim
  • net_ext stateful: false description: ""

[stgraber]

LXD uses the standard Go tar library, there's an issue upstream for support of sparse file. Looks like a fix was first merged in 1.10 (the next Go release) but had to be reverted, hopefully 1.11 will have a working version of this work.

https://github.com/golang/go/issues/13548

[stgraber]

Closing as since we're just using the standard compression library, as soon as this upstream issue is fixed, we'll automatically be handling sparse files correctly.

[jlec]

Thanks @stgraber for the quick triage.

[stgraber]

No worries, I was kinda hoping we were shelling out to tar for that part as it'd have made fixing this a bit easier, but no, since we need to rewrite some file attributes as we add them to the tarball, we've got to use the native golang implementation and so get bitten by this bug.