Upload archive to container fails with "expected EOF: unexpected EOF"

[BarDweller]

/kind bug Description

Podman won't accept tar archive content that docker will via the copy archive to container command.

Steps to reproduce the issue:

1. Build code from https://gist.github.com/BarDweller/8d774c52781a7c29bad67d02736bc246

2. launch podman system service with loglevel debug

3. run code from step 1 against unix socket from 2, passing id of an existing container

Describe the results you received: Archive is not written into container requested, following error is seen in the podman system service log.

Error seems related to Schrodingers EOF ;) error reading tar stream: expected EOF: unexpected EOF

ERRO[0007] failed to copy to the c867c0845083 container path /: error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/home/myuserid/.local/share/containers/storage/overlay/e115712143a816e41a02ee6ff8c51770d13fbbcde1b31315dc593b1cb408ffa0/merged", rootPrefix:"/home/myuserid/.local/share/containers/storage/overlay/e115712143a816e41a02ee6ff8c51770d13fbbcde1b31315dc593b1cb408ffa0/merged", Directory:"/", preservedDirectory:"/home/myuserid/.local/share/containers/storage/overlay/e115712143a816e41a02ee6ff8c51770d13fbbcde1b31315dc593b1cb408ffa0/merged", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(0xc0004b7930), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc0004b7940), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: error reading tar stream: expected EOF: unexpected EOF 

Describe the results you expected:

Archive is accepted & written to container.

Additional information you deem important (e.g. issue happens only occasionally):

Same archive is accepted by docker.

Output of podman version:

podman version 3.0.0-dev

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.0-dev
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: podman-3.0.0-1607105050.gitf01630ac.el8.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.7, commit: c79fee36146e1900149f7a32228f32614125c711'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.2"
  eventLogger: file
  hostname: myuserid.remote.csb
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 109953
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 109953
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 4.18.0-212.el8.x86_64
  linkmode: dynamic
  memFree: 590897152
  memTotal: 33402564608
  ociRuntime:
    name: runc
    package: runc-1.0.0-68.rc92.module+el8.3.1+8686+2a59bca3.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/109953/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.6-1.module+el8.3.1+8686+2a59bca3.x86_64
    version: |-
      slirp4netns version 1.1.6
      commit: a995c1642ee9a59607dccf87758de586b501a800
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 8192
  swapTotal: 16873680896
  uptime: 335h 56m 54.67s (Approximately 13.96 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/myuserid/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-2.module+el8.3.1+8686+2a59bca3.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/myuserid/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 5
  runRoot: /run/user/109953
  volumePath: /home/myuserid/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1607105057
  BuiltTime: Fri Dec  4 13:04:17 2020
  GitCommit: c79fee36146e1900149f7a32228f32614125c711
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 3.0.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.0-1607105050.gitf01630ac.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

@vrothberg PTAL

[vrothberg]

Can you try the latest master branch? The commit from your podman info does not exist (https://github.com/containers/podman/commit/c79fee36146e1900149f7a32228f32614125c711).

[BarDweller]

I'm running against master, I've tried pulling/rebuilding podman a few times since, with no change.

[vrothberg]

Quick update here since @BarDweller and I were chatting in private to speed things up.

It seems like the tar archive is slightly off in that it declares a directory as a file (in the associated tar header). Now tar and Podman do not like that but Docker does. As we have to remain compatible, I am opening a PR against Buildah to catch that case.

[vrothberg]

https://github.com/containers/buildah/pull/2845

[nalind]

I can't reproduce the "unexpected EOF". It looks like that first entry which you're attempting to name "/", to force the library to mark it as a directory, is being normalized by stripping off all initial "/" characters first, after which the library produces an entry for a regular file with a zero-length name. We've opened a PR to skip entries like that.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since this error is not reproducible, I am going to close. Please try it out against the main branch, or see if you can reproduce on a different machine.

[perklet]

This happens every time for python's cibuildwheel:

✓ pp39-manylinux_x86_64 finished in 18.59s
Copying wheels back to host...

Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.

                                                              ✕ 0.89s
Error: Command docker exec -i cibuildwheel-19a7fd5d-a550-4233-be7a-89cfd4c580cd tar -cC /output -f - . | tar -xf - failed with code 2. None

Podman version

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

[vrothberg]

@yifeikong could share an exact reproducer? I have no idea what cibuildwheel is but with a reproducer I may be able to track things down. There is a chance though that it may just work on my machine given the version of Podman (v3.4.4) you are using is a bit old.

[perklet]

@vrothberg Thanks for the pointer! I just updated podman to 4.x and it works. I was using the version coming from Ubuntu 22.04 and didn't know that it was a very old version.

[vrothberg]

Glad it's working. Thanks for checking and reporting back!