remote exec: empty output

[edsantiago]

Seen in remote gating tests:

not ok 200 [065] podman cp into a subdirectory matching GraphRoot
...
# [11:17:15.133378661] # /usr/bin/podman-remote cp /tmp/podman_bats.JxSzCS/cp-test-in/o0wChMA3RosZSJq6MqHr cpcontainer:/var/lib/containers/storage/o0wChMA3RosZSJq6MqHr
#
# [11:17:15.318381032] # /usr/bin/podman-remote exec cpcontainer ls -l /var/lib/containers/storage/o0wChMA3RosZSJq6MqHr
# [11:17:15.536517023] -rw-r--r--    1 root     root            51 Feb 19 11:17 /var/lib/containers/storage/o0wChMA3RosZSJq6MqHr
#
# [11:17:15.552084075] # /usr/bin/podman-remote exec cpcontainer cat /var/lib/containers/storage/o0wChMA3RosZSJq6MqHr
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: Contents of file copied into container
# #| expected: 'oklbEhcP2bFx9FvD34tHWFTUQuNwWSWgC081m6CKI8UymYXDgX'
# #|   actual: ''
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The ls clearly shows that the file exists and has the correct size.

I have not seen this in Cirrus. I'm stumped. This is probably not worth too much investigative effort -- yet -- but I'm filing because it's worth tracking.

Other info:

Packages:
  Kernel: 6.8.0-0.rc4.20240212git716f4aaa7b48.35.fc40.x86_64
  aardvark-dns-1.10.0-1.fc40.x86_64
  conmon-2.1.8-4.fc40.x86_64
  containers-common-0.57.3-3.fc40.noarch
  containers-common-extra-0.57.3-3.fc40.noarch
  container-selinux-2.229.0-2.fc40.noarch
  crun-1.14.3-1.fc40.x86_64
  iptables-legacy-1.8.10-7.fc40.x86_64
  iptables-legacy-libs-1.8.10-7.fc40.x86_64
  iptables-libs-1.8.10-7.fc40.x86_64
  libslirp-4.7.0-6.fc40.x86_64
  netavark-1.10.3-1.fc40.x86_64
  passt-0^20240216.g08344da-1.fc40.x86_64
  passt-selinux-0^20240216.g08344da-1.fc40.noarch
  podman-5.0.0~rc2-1.fc40.x86_64
  podman-debuginfo-5.0.0~rc2-1.fc40.x86_64
  podman-debugsource-5.0.0~rc2-1.fc40.x86_64
  podman-docker-5.0.0~rc2-1.fc40.noarch
  podman-remote-5.0.0~rc2-1.fc40.x86_64
  podman-remote-debuginfo-5.0.0~rc2-1.fc40.x86_64
  podmansh-5.0.0~rc2-1.fc40.x86_64
  podman-tests-5.0.0~rc2-1.fc40.x86_64
  python3-systemd-235-9.fc40.x86_64
  rpm-plugin-systemd-inhibit-4.19.1.1-1.fc40.x86_64
  slirp4netns-1.2.2-2.fc40.x86_64
  systemd-255.3-1.fc40.x86_64
  systemd-libs-255.3-1.fc40.x86_64
  systemd-networkd-255.3-1.fc40.x86_64
  systemd-oomd-defaults-255.3-1.fc40.noarch
  systemd-pam-255.3-1.fc40.x86_64
  systemd-resolved-255.3-1.fc40.x86_64
  systemd-udev-255.3-1.fc40.x86_64

...
# Arch:amd64 OS:fedora40 Runtime:crun Rootless:false Events:journald Logdriver:journald Cgroups:v2+systemd Net:netavark DB:sqlite Store:overlay

[Luap99]

This sounds like the same as in https://github.com/containers/podman/issues/8972 which well was never actually fixed.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

This one is cropping up again, both in int and sys tests. The interesting factors are that (1) no ginkgo retries on int tests, and (2) sys test failures are when run in parallel.

• fedora-39 : sys remote fedora-39 root host boltdb [remote]
  • PR #23275
  • 08-29 17:27 in [sys] |075| podman exec --tty
• fedora-40 : int remote fedora-40 root host boltdb [remote]
  • PR #23881
  • 09-05 16:29 in Podman checkpoint podman checkpoint and restore dev/shm content with --export and --import

x	x	x	x	x	x
sys(1)	remote(2)	fedora-39(1)	root(2)	host(2)	boltdb(2)
int(1)		fedora-40(1)