search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.6k stars 2.4k forks source link

Option to use compression in `podman image scp` #23192

Open cyqsimon opened 3 months ago

cyqsimon commented 3 months ago

Feature request description

When using podman image scp to transfer images between hosts, I noticed that the image is copied over the network as is without any compression:

`podman images | rg TAG`

Total network traffic reported by Bandwhich

However, if I do podman save <IMAGE> | zstd -o my-image.tar.zst, the compressed package often comes out to just about 1/3 the size:

Compression output with ZSTD

I can then transfer this file over the network manually and podman load -i my-image.tar.zst on the remote host.

Given the significant data savings, I find it odd that podman image scp does not support such a compression option.

Suggest potential solution

Add a -c|--compress option to podman image scp to perform compression and decompression on the fly, with optionally the ability to pick a compression algorithm.

Have you considered any alternatives?

podman save | zstd + podman load as described above, but it seems to me that there is no reason this couldn't be streamlined/automated.

Additional context

No response

rhatdan commented 3 months ago

Interested in opening a PR?

cyqsimon commented 3 months ago

Interested in opening a PR?

I would if I am proficient in Golang, which unfortunately I'm not. At least not yet. If anyone is interested in claiming this please go ahead.

afbjorklund commented 3 months ago

The "easiest" would be to add the compression flag (-C) to the scp command...

https://github.com/containers/common/blob/v0.59/pkg/ssh/ssh.go#L51

     -C      Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11, TCP and
             UNIX-domain connections).  The compression algorithm is the same used by gzip(1).  Compression is desir‐
             able on modem lines and other slow connections, but will only slow down things on fast networks.  The
             default value can be set on a host-by-host basis in the configuration files; see the Compression option.

When using the native (not golang) ssh, it should be possible to set in config?

github-actions[bot] commented 2 months ago

A friendly reminder that this issue had no activity for 30 days.

zackattackz commented 1 month ago

If no one else had one in progress, I'd be interested to try to submit a PR for this since I was also looking into #23297 which is similar.

rhatdan commented 1 month ago

It's all yours.

zackattackz commented 2 weeks ago

I have a WIP commit for this issue at #24211. It will need to be refined and split into several PRs, but I would appreciate any feedback from those interested here :)