Consider distributing statically built binaries as part of release

[ahmetb]

Currently if you're not using Fedora, you're kinda outta luck and you have to build from source. When I see a project written in Go, it's sad to see it just cannot be built directly with go tool, or doesn't offer binary builds.

Travis CI can attach binaries to your releases. For example: https://github.com/ahmetb/govvv/releases

You can use https://github.com/mitchellh/gox to provide cross platform builds in your travis build or so.

Please consider distributing binary builds.

[mtrmac]

If that is useful least for macOS (static linking device-mapper on Linux might not be a good idea), that would be nice. It would be much more likely likely to happen if someone contributed the automation to do this, though.

[git001]

I would be also very helpful got have some windows binaries as a lot of developer are still on windows.

[rhatdan]

Might have an intern look at this.

[jkroepke]

@rhatdan 10 months ago. Do you have news?

The static binary also works on any systems that go supports. Providing static builds for linux can resolve (https://github.com/containers/skopeo/issues/594), since the library link variant looks impossible to solve.

[mhagnumdw]

Any news here?

[mtrmac]

For Linux, there are now official containers, courtesy of @TomSweeneyRedHat ; see https://github.com/containers/skopeo/issues/432#issuecomment-634207980 . Still, the default recommendation is to use the build packaged in your distribution.

On macOS, similarly, there is a formula maintained by the Homebrew project.

I am not aware of any work on Windows.

[mtrmac]

For the record, there’s also #330 with a fair bit of overlap.

[seandilda]

Having official containers is great. Can we get official binaries as well so we can easily add them to our own containers (without having to pull them out of the other containers)?

I can install kubectl, helm, and oc all by using curl to pull in an official binary. Why is skopeo different?

[mtrmac]

Why is skopeo different?

https://github.com/containers/skopeo/issues/594#issuecomment-492827337

[jkroepke]

@mtrmac

Correct me if I'm wrong but https://github.com/containers/skopeo/issues/594#issuecomment-492827337 looks not related to this since the pure go static version (make binary-static DISABLE_CGO=1) does not include libdevmapper?

[mtrmac]

Well, that one is not handling NSS consistently with the rest of the system. None of them work well enough to be worth recommending over a distribution package.

[jkroepke]

@mtrmac

Well, that one is not handling NSS consistently with the rest of the system.

Could you explain this limitation a little bit more please? Resolution via /etc/hosts work well if compiled via make binary-static DISABLE_CGO=1.

A quick test:

bash-4.4# grep hosts.test /etc/hosts
127.0.0.4  hosts.test
bash-4.4# env GODEBUG=netdns=cgo+1 skopeo inspect docker://hosts.test/jkr/centos:7
go package net: using cgo DNS resolver
FATA[0000] Error parsing image name "docker://hosts.test/jkr/centos:7": error pinging docker registry hosts.test: Get https://hosts.test/v2/: dial tcp 127.0.0.4:443: connect: connection refused

For me, it looks like even if DISABLE_CGO=1 is set on compile time, a cgo dns resolver is available which using getaddrinfo. I guess getaddrinfo from libc should respect NSS.

I used go 1.13 for compilation. Here is the code how we compile skopeo: https://github.com/adorsys-containers/ci-helm/blob/70ed90496cc169c32652b6757e569fa249cfb0b6/images/3.2/Dockerfile#L8-L21

The DISABLE_CGO=1 version of skoepo works fine. I use skoepo multiple times in different CI/CD pipelines on different projects. None of them had issues.

None of them work well enough

What not working well enough? Could you explain this more? I would like to understand. I didn't see the limitations.

Edit:

It looks like the make binary-local DISABLE_CGO=1 command doesn't build a pure static build:

bash-4.4# ldd /usr/local/bin/skopeo
    linux-vdso.so.1 (0x00007ffdffa73000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe910e09000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe910a47000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe912a48000)

[mtrmac]

nss_sss? nss_myhostname? Any other arbitrary NSS module? (This is impossible in a statically-linked binary because glibc explicitly does not have a stable NSS module API.)

[jkroepke]

You are right.

But maybe it's okay. Is nss inside a container a case? Could you explain if this is a real world issue, why so many golang project provide a static build? docker? kubectl? helm? reg? cri-o?

Again, why is skopeo different?

Also mac and windows users could also benefit from this. I know community installation methods and some of the only allows to install the lastest available release. But what the reason for not provide official one?

I think it's still worth that skopeo distributes such binaries on released version. A lot of other do this, too. It would be a great benefit if I could like @seandilda use just curl to install skopeo inside my docker image.

[rhatdan]

Can you just use the nix version?

[mtrmac]

But maybe it's okay. Is nss inside a container a case?

It’s a fair point that the container does not integrate with outside configuration either.

Could you explain if this is a real world issue, why so many golang project provide a static build? docker? kubectl? helm? reg? cri-o?

Again, why is skopeo different?

I guess they prioritize virally-easy installation over correctness in corner cases. That’s a legitimate choice.

Why is using a distribution package not the most convenient option anyway? I guess you are probably already using the package installation tools, it requires even less set-up and typing, and makes inspection/maintenance/updates possible.

Also mac and windows users could also benefit from this.

That’s separate #330. (Mac and Windows do, at least to an extent, provide a stable user-space ABI, so we would not have to — and indeed couldn’t! – provide statically-linked builds.)

[jkroepke]

Why is using a distribution package not the most convenient option anyway?

A distribution package is also a good choice, but:

Long time I used the package from centos 7 but the package get outdated and rarely updated. Right now the skoepo package on centos 8 is 0.1.40.

[root@739da533f0a5 /]# yum info skopeo
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:16 ago on Tue Aug 11 20:03:48 2020.
Available Packages
Name         : skopeo
Epoch        : 1
Version      : 0.1.40
Release      : 11.module_el8.2.0+377+92552693
Architecture : x86_64
Size         : 5.8 M
Source       : skopeo-0.1.40-11.module_el8.2.0+377+92552693.src.rpm
Repository   : AppStream
Summary      : Inspect container images and repositories on registries
URL          : https://github.com/containers/skopeo
License      : ASL 2.0
Description  : Command line utility to inspect images and repositories directly on Docker
             : registries without the need to pull them

The package on alpine looks good.

For Ubuntu or RHEL UBI I can't find such a package.

Looking at the documentation, I could use a 3rd party repo for debian or rhel based systems to install the lastest version of skopeo ...

A counter part for using packages inside docker is the size of the docker image. If I install a package, the package db would be altered (new layer in docker images). It's also common that such packages install unnecessary files (like man pages, bash-completion) or dependencies (for example a containers-common package is installed while installing skopeo from the Kubic repository).

An other gap compared to attached binaries on a github release is the version again. There is only one version of skopeo available inside the packages repository. For example if the new version have a regression, my container build/images just gets broken because I could only install the lastest version of skoepo (or depends on the linux distro a very old stable one, but from time to time I would miss features).

Currently I could pin the version of skopeo. If there is a new version of skopeo I could test the version if everything is okay in our environment. After that I could publish the image. Thats possible because I build skoepo from source.

An example: https://github.com/adorsys-containers/ci-helm/blob/70ed90496cc169c32652b6757e569fa249cfb0b6/images/3.2/Dockerfile#L10

If you take a look below the skopeo build you would side a lot of docker ARG (define the version) and curl to install all the other packages.

I know that using curl from somewhere is more risky then a well defined package repository (except its a 3rd party one) but I'm trust https and github as source that the binary are not modified.

[rhatdan]

Is the NIX static version of Skopeo enough to satisfy this issue?

[jkroepke]

We used skopeo only for copy images from docker registry a to docker registry. Like a pipe in a shell.

We are doing this multiple times across the while company and our customers each day since more then one year. It including copies images from or to a gitlab registry, harbor registry, docker hub and a openshift registry. It fits perfect.

We had no issues with skopeo but I could only say this for the copy images feature.

Thanks for such a stable tool!

[TomSweeneyRedHat]

@jkroepke thanks for the update! I'm curious, are you using an installed version of Skopeo, or one of the container images such as quay.io/containers/skopeo:latest?

[jkroepke]

I'm compiling skopeo from source using make binary-local DISABLE_CGO=1

See: https://github.com/adorsys-containers/ci-helm/blob/bd94c07ce719ee4873f751e7b29d982d9406c9df/images/3.3/Dockerfile#L19-L21

[rhatdan]

Ok since we don't want to say we "support a static" build, we allow users to build it, and have the nix version available as well as skopeo inside of a container. I think that answers this issue.

[jkroepke]

Recheck if u really doesn’t want to support it. Users will constantly re ask for it.

Go for a slim variation which drops all features that are not supported on static bulids. No one needs the devicemapper capabilities of skopeo.

If someone depends on static builds, I recommend to switch to crane

[seandilda]

I'd like to reiterate what @jkroepke said. For us, skopeo is one of several tools that gets copied onto an image used for most jobs in our CI/CD pipeline. The sole job of skopeo for us is to copy the image from our CI docker registry into our production OKD registry.

Our jobs aren't currently setup to have the skopeo copy as a separate job than should use a separate container. I've never used Nix, but it looks like a whole environment just to install a single binary, which seems a bit overkill.

[joaompinto]

@rhatdan , I believe the argument used to close this issue is poorly supported and leaves open questions about the effective interest of this project in reaching a wider audience of open source users.

Most people which would benefit from Skopeo are fully aware of the limitations of using static binaries, it is not a new "problem" and it is actually a quite common approach on the containers tool world.

Is there a significant effort on building and providing a static binary ? Are we defending that this open source software software project should be restricted to use on Linux distributions which have the ability to build/maintain it ? Are we defending that we should use a specific package manager (Nix) as an alternative to static binaries in Linux ?

Thanks

[joaompinto]

Also an example on where I believe the right decision was done:

https://github.com/containers/crun/issues/176

[mtrmac]

Note that the Nix references are not currently relevant — that implementation was dropped when the original contributor abandoned that approach.

And that’s basically where we are at — Makefile contributions to build statically would be gladly accepted, but it would also be something we don’t use, probably won’t notice if it breaks, and won’t spend much effort to fix.

---

Are we defending that this open source software software project should be restricted to use on Linux distributions which have the ability to build/maintain it ?

I’m certainly sheltered in my little world, but is that a significant problem? My impression is that there aren’t that many distributions. Or is the concern about running major distributions, but on old stable, or even already unmaintained, versions, and nevertheless wanting a new Skopeo build?

[mtrmac]

FWIW current version of this discussion is https://github.com/containers/skopeo/issues/1478 .

[joaompinto]

I’m certainly sheltered in my little world, but is that a significant problem? My impression is that there aren’t that many distributions. Or is the concern about running major distributions, but on old stable, or even already unmaintained, versions, and nevertheless wanting a new Skopeo build?

The term significant is quite subjective, more than one distribution is enough of a blocker if they adopt different maintenance policies regarding major versions Having one of the few new, stable, and maintained distributions does not insure you get the skopeo version that you might need for a specific purpose which is different of the intended by the Linux distribution package updates policy.

On my specific case I would like to provide a integration which relies on Skopeo, and I would prefer no to depend on the distributions packaging policy and/or 3rd party repos availability.

I will follow the progress in #1478, thanks for the feedback.

[sunshine69]

Just tiring of some errors and comes here.. While the author does not like the static binary support I would question why, but some of the reason might not sound enough for me to wholy drop the philosophy of simplicity of deployment of golang.

Like all k8s tool outthere they are very successfull ....

• Why this tools requires gpgme (process gpg, is ther any native golang doing so?)
• Why it requires livdevice-mapper - is it not enough to use the file system exposed by the kernel itself?
• etc .. stop here

I personally do not think that approach which drop or not supporting static binary due to the CGO enabled is a good direction.

Because it says not supported so the static build command stuck with errors, I dare not to create ticket for it, however if anybody still care, as of now ...

 git clone git@github.com:containers/skopeo.git
Cloning into 'skopeo'...
Warning: Permanently added 'github.com,13.236.229.21' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 34847, done.
remote: Counting objects: 100% (593/593), done.
remote: Compressing objects: 100% (435/435), done.
remote: Total 34847 (delta 183), reused 296 (delta 115), pack-reused 34254
Receiving objects: 100% (34847/34847), 27.08 MiB | 5.49 MiB/s, done.
Resolving deltas: 100% (19872/19872), done.
stevek@macbook-work 11:38 ~/src> cd skopeo/
stevek@macbook-work 11:38 ~/s/skopeo main> docker run -v $PWD:/src -w /src -e CGO_ENABLED=0 golang \
                                           make BUILDTAGS=containers_image_openpgp GO_DYN_FLAGS=
Unable to find image 'golang:latest' locally
latest: Pulling from library/golang
0c6b8ff8c37e: Pull complete 
412caad352a3: Pull complete 
e6d3e61f7a50: Pull complete 
461bb1d8c517: Pull complete 
9297634c9537: Pull complete 
6a0b51344be5: Pull complete 
856234c9f319: Pull complete 
Digest: sha256:301609ebecc0ec4cd3174294220a4d9c92aab9015b3a2958297d7663aac627a1
Status: Downloaded newer image for golang:latest
CGO_CFLAGS="" CGO_LDFLAGS="" GO111MODULE=on go build -mod=vendor  -ldflags '-X main.gitCommit=13cd098079aaaa668219e581b61dd4ac5c8ad614 ' -gcflags "" -tags "containers_image_openpgp" -o bin/skopeo ./cmd/skopeo
sed -e 's/\((skopeo.*\.md)\)//' -e 's/\[\(skopeo.*\)\]/\1/' docs/skopeo-copy.1.md  | /go/bin/go-md2man -in /dev/stdin -out docs/skopeo-copy.1
/bin/sh: 1: /go/bin/go-md2man: not found
make: *** [Makefile:147: docs/skopeo-copy.1] Error 127
stevek@macbook-work 11:40 ~/s/skopeo main> 

This is the end of it. I run Ubuntu 20.04 , no package. I did get into the thrid party to support and more than 5 mins not getting where, what I want.

[mtrmac]

That failure has nothing to do with static builds, and how to handle that is documented in the installation documentation.

Now, I totally get that reading such documentation can not be generally expected — but in my mind, that’s again a reason to prefer distribution packages, where “reading installation documentation” is not a necessary step.