Create releases of sops that does not require users to install go

[jessechahal]

Without some way of releasing SOPS in easily installable packages it starts to feel more like an experiment rather then a fully fledged application. Requiring that users install and setup Go seems unnecessary burden especially since go can compile to native binaries.

• Have packages -- OSX: macports or homebrew packages, maybe even .pkg files at some point -- RHEL/CENTOS: have RPM packages -- Debian/Ubuntu: have apt packages -- Windows: msi installs

In the meantime maybe having automated github releases of binaries with installation instructions would be a good stopgap solution. These could be built on travis and pushed directly to github.com/mozilla/sops automatically

[autrilla]

We currently have a brew package for sops, but it's for the Python version. It takes a while for the brew people to process updates, but I'm sure we'll eventually have go-sops on brew. I'm certainly in favor of having build artifacts on circleci pushed to S3 or some other place so you can just download a static binary and not compile locally, and I'd also be in favor of compiling packages for the most popular package managers as part of our build process (although perhaps only for the master branch).

We do not support windows, as far as I know. I haven't even tried if sops works there.

[jessechahal]

Is the plan to switch to circleci from travis at some point? I've never used circleci so I don't really have any comments regarding it. I'm personally not a windows person either. I just added it to the list for completeness as well as the fact that golang can compile to .exe files. Windows10 comes with a package manager that accepts chocolaty style package as well as a slightly newer Windows10 style packages (if msi files aren't desired). I don't think it would be an extensive rewrite in order to get windows powershell support working although I don't think circleci or travis have windows support from a testing perspective -.-

[autrilla]

Oh, sorry, when I said circleci I actually meant travis. It's just that we also have some projects on circle and I mixed them up!

Yeah, travis and circle don't support Windows. I know the servo project uses https://www.appveyor.com/, so I suppose it's the best option for Windows CI. It's worth noting, that at least for servo, appveyor was around an order of magnitude slower than travis.

[jvehent]

I fail to understand how go get feels more experimental than pip install or brew install. All three are custom package managers built to replace the standard package managers provided by operating systems. Unlike brew, go get at least works across platforms.

That said, we could build static binaries and host them on S3, but I'd argue that people likely to use sops will also use a lot of tools built in Go, and probably have, or should have, Golang installed.

Is this not a reasonable assumption?

[jessechahal]

Python comes standard on most *nix distributions, golang does not. get get and pip install are comparable from a package manager perspective (although I wouldn't classify brew install as the same) as they are both package managers used with specific languages. From my perspective one of the main benefits of using golang over python was the ability to compile static binaries that do not require external dependencies. In our case we wish to run SOPS on servers and not just local dev workstations. Installing golang, pulling sops source code from github, pulling other external dependencies in a source code, compiling the code, and finally running the binary on a server (especially for a language that can compile to binaries) is not what I would call good practice. Also what happens in situtions where a server only whitelists certain domains, asking operations teams to whitelist github.com in order for software to work on a server is going to be an uphill fight (and one that will end up being lost by devs).

go get would make perfect sense if SOPS was purely a library and not a CLI tool. pip install is required for python packages and makes sense because the user MUST have python to run the tool anyways as python is an interpreted language. Ruby has the same problem. There are tools written such as virtualenv just to help get around some of the odd issues that come up with this configuration. Some software written with these languages even ship an embedded version of python or ruby just to get around the headache.

[jessechahal]

I think the current goto example of an extremely successful golang application is docker. Many people do not even know that a large portion of docker is written with golang and even if people did know it wouldn't matter because you do not need golang in order to run docker.

[jvehent]

Docker actually manages package repositories for various distributions. Thats a lot of infrastructure to setup and maintain. I think we could build packages for various systems using fpm and distribute those on s3, but whole repositories are a bit too much.

On Nov 7, 2016 19:46, "Jesse Chahal" notifications@github.com wrote:

I think the current go to example of an extremely successful golang application is docker. Many people do not even know that a large portion of docker is written with golang and even if people did know it wouldn't matter because you do not need golang in order to run docker.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mozilla/sops/issues/166#issuecomment-259011148, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZXgbxAECG9TQure6IyHR8M4ml3NFk9ks5q78ZYgaJpZM4KqXSh .

[jessechahal]

Sorry I wasn't trying to suggest trying to compete with docker in terms of infrastructure :P just meant that expecting people to have golang installed is not a good idea. FPM to S3 would be awesome!

[jvehent]

Hydra seems to have this automated in travisci: https://github.com/ory-am/hydra/blob/master/.travis.yml

[mrkschan]

FYI, Elastic Beats cross compile on TravisCI: https://github.com/elastic/beats/blob/master/.travis.yml

And, Windows compilation on AppVeyor: https://github.com/elastic/beats/blob/master/.appveyor.yml

They use gox to crosscompile - https://github.com/elastic/beats/blob/master/libbeat/scripts/Makefile

[jvehent]

How about this? https://go.mozilla.org/sops/dist/ No MacOS dist yet, because building DMGs from linux is tricky. If anyone has a solution, I'm all ears.

[jvehent]

I'll leave this open in case someone is motivated enough to add msi and dmg generation to this. From my point of view, rpm and deb solve the use case, other systems can use go get.

Note that dmg creation is complicated because OSX requires package signing using a certificate issued by Apple. We would have to pay for one, then use it safely in travisci.

[rouge8]

I've seen some other projects just provide binaries for all platforms (e.g. on GitHub releases) rather than packages.

[jvehent]

We could cross-compile and attach the binaries to the release. Does anyone have experience automating that process so it can run in CI?

[mikesimons]

I'd love to see binaries for sops as github releases because we want to use it for servers too.

Go makes cross platform builds crazy easy as long as you aren't using cgo and travis supports publishing release artifacts.

I automated this once:

• https://github.com/mikesimons/pacyak/blob/master/.travis.yml
• https://github.com/mikesimons/pacyak/blob/master/Makefile

Happy to reimplement for sops but would need someone from the project to contribute the secrets bit.

[rmb938]

Do we have any updates on this? I am currently trying to get sops inside an alpine docker container and I rather not install go then have to remove it.

[jvehent]

@rmb938 : can you use the packages from https://go.mozilla.org/sops/dist/ ? If anyone wants to submit a PR to publish artifacts from travisci, I'm interested.

[rmb938]

Those packages are only compatible with rhel and debian based os' not alpine. A pr can be made but it wont work as the owners of this repo need to be the ones that provide the secrets.

[autrilla]

If all you want is the binary in a container, you can just do something like this:

FROM golang:1.7

RUN go get go.mozilla.org/sops/cmd/sops
RUN CGO_ENABLED=0 GOOS=linux go install -a -ldflags '-extldflags "-static"' go.mozilla.org/sops/cmd/sops

FROM alpine:latest
WORKDIR /root/
COPY --from=0 /go/bin/sops .
CMD ["./sops"]

[jvehent]

This is now fixed, packages and binaries are in https://github.com/mozilla/sops/releases/tag/2.0.10