Podman import xz compressed rootfs can not save and load to the other machine

meilier commented 1 year ago

Issue Description

podman import xz compressed rootfs can not save and load again.

Steps to reproduce the issue

1. podman import xxx.tar.xz test:latest
2. podman save test:latest -o test.tar
3. podman rmi test:latest
4.   podman load -i test.tar

Describe the results you received

[root@localhost podmanupload]# podman load -i test.tar
Getting image source signatures
Copying blob b4bff416429f [=====================================>] 4.8MiB / 4.9MiB
Error: payload does not match any of the supported image formats:
 * oci: initializing source oci:test.tar:: open test.tar/index.json: not a directory
 * oci-archive: loading index: open /var/tmp/oci3545856028/index.json: no such file or directory
 * docker-archive: writing blob: storing blob to file "/var/tmp/storage3899402231/1": happened during read: Digest did not match, expected sha256:b4bff416429f4b429432f77a9d2395e9cf89f3aa38a5ebe37b7097bfecd3e7b3, got sha256:ce8792ade25e61d4677b53f88bda40ef24473c1c8e3f2420b0906471d017d8e7
 * dir: open test.tar/manifest.json: not a directory
[root@localhost podmanupload]#

Describe the results you expected

import success

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/bin/conmon
    version: 'conmon version 2.1.7, commit: 9a3af8f438d89ffc176f3c76c164088e64bb6b19'
  cpuUtilization:
    idlePercent: 96.73
    systemPercent: 0.62
    userPercent: 2.65
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: '"openEuler"'
    version: "22.03"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.12
  linkmode: dynamic
  logDriver: journald
  memFree: 1199435776
  memTotal: 7797182464
  networkBackend: cni
  ociRuntime:
    name: runc
    package: containerd.io-1.6.18-3.1.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      commit: v1.1.4-0-g5fd4c4d
      spec: 1.0.2-dev
      go: go1.19.6
      libseccomp: 2.5.3
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-1.oe2203sp1.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 5832261632
  swapTotal: 8472489984
  uptime: 352h 43m 20.00s (Approximately 14.67 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 64378073088
  graphRootUsed: 44625469440
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.0-dev
  Built: 1680837502
  BuiltTime: Fri Apr  7 11:18:22 2023
  GitCommit: a462b37703999185248bc4962be5abc754f99c0d
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0-dev

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

Try to seach and locate, it seems like wrong compression algorithm is used in containers/image, result in a wrong diffID saved.

But when I try decompress it, import becomes very slow, there may be too many decompressiones during import.

https://github.com/containers/image/blob/main/tarball/tarball_src.go#L88

vrothberg commented 1 year ago

Thanks for reaching out, I can reproduce the issue, @meilier.

No problem with a simple tar but with an xz-compressed one.

Cc: @mtrmac

mtrmac commented 1 year ago

Yes, the tarball_src.go link is accurate: only uncompressed and gzip-compressed inputs could ever have worked.

Given that the man page has been documenting Bzip (not even Bzip2) and Xz since 2017, that’s before Podman was named Podman, I have to wonder whether we shouldn’t just stop advertising this in the documentation instead of implementing this.

There are two aspects to supporting Xz:

Instead of assuming gzip, detect the relevant compression, and compute DiffID correctly. Easy enough.
The somewhat awkward part: OCI doesn’t actually define any MIME type for Bzip{,2} nor Xz. IIRC it will actually work fine if we claim Gzip, but it will trigger a warning in https://github.com/containers/image/blob/bb66acc37f166c03470cb58d3b8c808c288e1c2a/copy/compression.go#L65 . The totally clean approach is for the tarball: transport to generate an uncompressed layer with an uncompressed MIME type, probably by using a temporary file in /var/tmp. That’s costly enough that it makes sense to me to recommend to users not to use Xz in the first place — using Gzip (or, assuming we add support, Zstd) would be faster, assuming the CPU can decompress faster than the disks can write.

So, given the 2017 history, and the awkwardness of truly and cleanly supporting Xz, I’m a bit tempted to declare this, or at least the Bzip/Xz parts, a documentation bug; and to only consider adding Zstd support.

OTOH implementing Xz, either naively in a way that triggers the warning, or correctly with a temporary file, would not really be that much work — it’s mostly a question of need/prioritization.

vrothberg commented 1 year ago

I'm OK with changing docs instead.

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@meilier Interested in opening a Docs PR?

jpalus commented 1 year ago

I'd argue that it shouldn't consider docs only. I'm ok with no support for xz/bzip2 but I think podman import should issue error then just like for any other unsupported format. Currently it gives false impression that it all works just fine. For example I've been using images imported and published from tar.xz in podman for years until it turned out it doesn't actually work in docker only because single hash is wrong (#18716).

mtrmac commented 1 year ago

I'm ok with no support for xz/bzip2 but I think podman import should issue error then just like for any other unsupported format.

That’s a fair point, yes; the tarball transport in c/image should reject images with a non-gzip compression, and Podman can just update documentation.

containers / podman