Release asset naming

[sagikazarmark]

Currently release asset names look like this: protolock.20180621T191216Z.darwin-amd64.tgz

It's quite hard to build tooling around this naming convention. It would be nice to have the version in the file name instead of the build date.

WDYT?

[nilslice]

@sagikazarmark - good thinking.. I would be happy to consider any changes you propose. Unless the timestamp is specifically needed by others, I think a version number would suffice in its place.

cc/ @noel-yap for guidance & recommendation as well.

[noel-yap]

AFAIK, there have been no actual releases but now it occurs to me that I may not have been looking in the right place for any released artifacts. Are there publicly published released artifacts? If so, where are they?

If there are, IMO, it would make sense to follow https://semver.org/ and even https://keepachangelog.com/en/1.0.0/ and to document the release process (including publishing to some public repository).

And having said that, I may still have some invalid assumptions as to how Golang projects work WRT releases and published artifacts (my more recent experience has been in the Java world) so please call out these assumptions if this is the case.

[nilslice]

@noel-yap I believe this is in reference to the way the distributed pre-built binaries are named:

https://github.com/nilslice/protolock/releases

[noel-yap]

Yes, I understand that. What are the steps for those binaries to be published there? Can, for example, someone insert malicious source, build it, then publish the binaries to that location without committing the code?

Is there something like Maven Central or Bitbucket for Golang? Currently, we're building the binaries on premises and publishing to a private Artifactory instance.

I didn't get much sleep last night so I'm probably not communicating well my concerns and how they relate to the versioning scheme. I'll try to break it down more for my sake so I'm hoping no one takes it as me talking down to them. Also, my background having been in release engineering might help put things into context.

Ideally, versions mean something (eg SemVer) and represent the exact same bits to everyone. This means that one can checkout a git tag or branch representing the release (eg release/v0.1.0) and produce the "same" binary as what's been published. This is why I consider it important to document the release process and to have more confidence that it can't be easily subverted.

When I see timestamps instead of version numbers, it's an indication to me that the inputs to the release can change (ie they're not hermetic) such that checking out a specific release branch and building may not actually reproduce what's been published publicly. The timestamps also, at least in theory, can be be used to figure out the actual inputs to the build (including the dependencies that were used).

If I'm still not communicating clearly enough my concerns, I'll try to get better sleep tonight and try again tomorrow.

@sagikazarmark , what sorts of issues are you running into WRT tooling?

[nilslice]

@noel-yap this is great feedback -- and it's fantastic to have someone with release engineering experience contributing to the project (I admittedly have very little personal experience in that domain).

What are the steps for those binaries to be published there? ...

Currently, the only process is manually running the release.sh script locally to cross compile. I have been stringent to use the same commit to build the binaries as the tag (version used for the release) is pointing to.

Is there something like Maven Central or Bitbucket for Golang? Currently, we're building the binaries on premises and publishing to a private Artifactory instance.

GitHub often is used as a place to host and serve downloads of pre-built binaries, but I am not familiar with any CI/build process that would integrate directly with it in an automated, remote "test -> build -> release" cycle.

As per a request from the @envoyproxy team: https://github.com/envoyproxy/envoy/pull/3703#discussion_r197842287, a SHA verification process may end up helping solve some of the issues you bring up. I'm not familiar with this process though, do you have any suggestions?

[noel-yap]

I had thought of generating a SHA file but it occurred to me that it has the same issue. One can make a file change, build the artifact and the SHA file, publish them, and not commit/push the code change.

I want to be clear that I'm not implying you can't be trusted (I don't think you're reading into this this way but want to be sure). It sounds to me that you understand these processes are put in place also to guard against mistakes (eg one could forget to commit a code change, build out of the wrong area, etc).

There are a few things that need beefing up, IMO. You had mentioned automated CI and release. That's some of them. Another is ensuring the build is repeatable. This means not pulling dependencies at HEAD. I've heard other Golang projects would include the source of their dependencies as part of the project so that that wouldn't happen. I would rather find some way to use git submodules for this but I've heard it's not trivial.

If those things were put in place, I'd be all for using actual versions for this project. I'd investigate it myself but currently I have other priorities. @sagikazarmark , since you're feeling the most pain with the consequences of this, would you be able to do some legwork?

[nilslice]

@noel-yap this might be a solution for the automated build and release: https://goreleaser.com/

[sagikazarmark]

what sorts of issues are you running into WRT tooling?

Knowing the version number and the URL scheme is usually enough to just upgrade the version in a script or a dockerfile. Having a timestamp in the URL makes that impossible without manually checking the releases page every time.

This means not pulling dependencies at HEAD. I've heard other Golang projects would include the source of their dependencies as part of the project so that that wouldn't happen. I would rather find some way to use git submodules for this

Golang has nice dependency management solutions that can lock to exact commits of other dependencies. Personally I don't like including the dependencies in the VCS when we have the tools to properly manage the dependencies. Since they are mostly third-parties anyway, I don't see the reason not to trust them when installing dependencies. If someone tampers with the version tags the dependency tool will either install it's locked version or complain that it's missing.

Goreleaser is a popular tool for building Golang CLI tools, I used it a few times.

since you're feeling the most pain with the consequences of this, would you be able to do some legwork?

What legwork do you exactly mean?

Here are a few examples for automated, pre-built binary releases:

https://github.com/golang/dep/releases https://github.com/twirphp/twirp/releases

[noel-yap]

The legwork of putting in place a good release process for protolock. I don't know enough to choose among the various tools, etc. It sounds like you and Steve are in a better position to put the process into place.

On Fri, Jun 29, 2018 at 3:56 PM Márk Sági-Kazár notifications@github.com wrote:

what sorts of issues are you running into WRT tooling?

Knowing the version number and the URL scheme is usually enough to just upgrade the version in a script or a dockerfile. Having a timestamp in the URL makes that impossible without manually checking the releases page every time.

This means not pulling dependencies at HEAD. I've heard other Golang projects would include the source of their dependencies as part of the project so that that wouldn't happen. I would rather find some way to use git submodules for this

Golang has nice dependency management solutions that can lock to exact commits of other dependencies. Personally I don't like including the dependencies in the VCS when we have the tools to properly manage the dependencies. Since they are mostly third-parties anyway, I don't see the reason not to trust them when installing dependencies. If someone tampers with the version tags the dependency tool will either install it's locked version or complain that it's missing.

Goreleaser is a popular tool for building Golang CLI tools, I used it a few times.

since you're feeling the most pain with the consequences of this, would you be able to do some legwork?

What legwork do you exactly mean?

Here are a few examples for automated, pre-built binary releases:

https://github.com/golang/dep/releases https://github.com/twirphp/twirp/releases

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nilslice/protolock/issues/43#issuecomment-401493571, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWU8WYAeGjQb3NYeM6ju7a-c79WhuAqks5uBrCQgaJpZM4U2Mof .

[nilslice]

I'd like to investigate GoReleaser for the automated releases. I just need to ensure that there is a checksum option to satisfy the needs of this PR: https://github.com/envoyproxy/envoy/pull/3703#discussion_r197842287

@sagikazarmark - if you have some tips on integration, please let me know. Also, to get back to your original question, I think that if we just replace the timestamp with the release tag, it will get you what you want while still maintaining the immutability promise of the release (semi-provided by the timestamp). Do you agree? Please let me know if there are other suggestions you would make. @noel-yap if you disagree with this, let me know as well. I think it would be reasonable to support others desire to build tooling around the ability to download a release by using a URL template of: $GITHUB/nilslice/protolock/releases/download/$TAG/protolock.$TAG.$OS-$ARCH.tgz

What do you think?

[noel-yap]

I'm OK with this so long as there's an audit trail from the binary back to the source.

[noel-yap]

FWIW, https://github.com/npm/npm/issues/21202 is the kind of stuff I would like the process to prevent.

On Fri, Jun 29, 2018, 17:01 Steve Manuel notifications@github.com wrote:

I'd like to investigate GoReleaser for the automated releases. I just need to ensure that there is a checksum option to satisfy the needs of this PR: envoyproxy/envoy#3703 (comment) https://github.com/envoyproxy/envoy/pull/3703#discussion_r197842287

@sagikazarmark https://github.com/sagikazarmark - if you have some tips on integration, please let me know. Also, to get back to your original question, I think that if we just replace the timestamp with the release tag, it will get you what you want while still maintaining the immutability promise of the release (semi-provided by the timestamp). Do you agree? Please let me know if there are other suggestions you would make. @noel-yap https://github.com/noel-yap if you disagree with this, let me know as well. I think it would be reasonable to support others desire to build tooling around the ability to download a release by using a URL template of: $GITHUB/nilslice/protolock/releases/download/$TAG/protolock.$TAG.darwin-amd64.tgz

What do you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nilslice/protolock/issues/43#issuecomment-401501538, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWU8bIyvkTNmRSUToB99kjpfG2JiqI9ks5uBr_dgaJpZM4U2Mof .

[sagikazarmark]

Sorry, I lost in the jungle of Github emails.

Having the tag in the file name is okay, actually a good practice. The proposed format looks good to me. 👍

[sagikazarmark]

This should work for CircleCI and Goreleaser:

https://github.com/banzaicloud/banzai-cli/blob/master/.circleci/config.yml https://github.com/banzaicloud/banzai-cli/blob/master/.goreleaser.yml

[nilslice]

Awesome -- will dig in once I'm back from vacation. Thanks!