search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.09k stars 3.2k forks source link

Support generating protobuf & grpc service defs from argo packges #853

Closed gaigepr closed 5 years ago

gaigepr commented 6 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

This is a FEATURE REQUEST. At present, I can successfully use proteus to generate a .proto from argoproj/argo/pkg/apis/workflow/v1alpha1 but there are a few problems left to solve.

The various k8s.io packages that argo imports must also have .proto files generated if the .proto generated for argo itself is going to be a complete representation of the go definitions. This has not worked so far in my experiments. Often proteus segfaults when trying to run on k8s.io packages and other times it generates mostly empty .proto files.

You can try too:

$ pwd
/home/user/go/src/github.com/argoproj/argo/vendor
$ proteus -f ../proto -p k8s.io/api/core/v1 --verbose
...

Take a look inside /home/user/go/src/github.com/argoproj/argo/proto for what it outputs, if it even runs...

Here is what I have been doing to generate .proto from argo's workflow package.

$ pwd
/home/user/go/src/github.com/argoproj/argo
$ proteus -f ./proto -p github.com/argoproj/argo/pkg/apis/workflow/v1alpha1 --verbose

This produces a .proto file under ./proto/github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto which is great! It also generates a pkg/apis/workflow/v1alpha1/generated.pb.go.

However, there are a few problems. Not all the messages have all their fields. For example,

v1alpha1.Item is defined as follows:

// Item expands a single workflow step into multiple parallel steps
// The value of Item can be a map, string, bool, or number
type Item struct {
    Value interface{}
}

But the resulting message lacks any fields... This has to do with an incompatibility between go's interface{} type and protoc (including gogo/protoc). One solution might be to use the Any proto type from google. See this stackoverflow thread.

// Item expands a single workflow step into multiple parallel steps
// The value of Item can be a map, string, bool, or number
message Item {
    option (gogoproto.goproto_getters) = false;
    option (gogoproto.typedecl) = false;
}

How to reproduce it (as minimally and precisely as possible):

  1. install proteus
  2. run the following from the root this project: 
    mkdir proto && \
proteus -f ./proto \
    -p github.com/argoproj/argo/pkg/apis/workflow/v1alpha1 \
    --verbose
  3. Inspect the contents of ./proto

For what its worth, I also tried using go-to-protobuf and had essentially 0 success.

Environment:

gaigepr commented 6 years ago

Another issue related to this, proteus and go-to-protobuf doesn’t seem to know how to generate proto import statements. This is part of why running proteus with --verbose shows so many wanrings about ignoring certain types because their proto def could not be found.

jessesuen commented 6 years ago

@gaigepr -- I created a branch which changes the definition of the Workflow Item such that it is no longer an interface{}. The change is here:

https://github.com/jessesuen/argo/tree/protobuf

You can give that a try to see if it helps.

gaigepr commented 6 years ago

@jessesuen -- Here are some notes on my attempts:

I cloned your fork and checked out the protobuf branch. Painless! Proteus is installed from my previous attempts so I did not have to do that.

$ pwd
/Users/gaige/go/src/github.com/jessesuen/argo
$ mkdir proto
$ proteus -f ./proto -p github.com/jessesuen/argo/pkg/apis/workflow/v1alpha1 --verbose
# no output, This usually means nothing happened
$ ls proto
# no output, the dir is empty.
$ proteus -f ./proto -p k8s.io/api/core/v1 --verbose
# no output
$ ls proto
# no output

Interestingly, if I do the following, I get proto generated but of course since it is generated from argoproj/argo your changes to Item hasn't been applied.

$ proteus -f ./proto -p github.com/argoproj/argo/pkg/apis/workflow/v1alpha1 --verbose
WARN: ignoring type interface{}
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ContainerPort" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvFromSource" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvVar" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ResourceRequirements" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeMount" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeDevice" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Lifecycle" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "TerminationMessagePolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "PullPolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecurityContext" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ContainerPort" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvFromSource" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvVar" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ResourceRequirements" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeMount" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeDevice" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Lifecycle" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "TerminationMessagePolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "PullPolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecurityContext" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Affinity" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Container" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Toleration" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "UID" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/types will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "OwnerReference" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Initializers" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Volume" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "PersistentVolumeClaim" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Affinity" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Toleration" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "LocalObjectReference" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Volume" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
INFO: Generated proto: proto/github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto
INFO: executing protoc: /usr/local/bin/protoc --proto_path=/Users/gaige/go/src:./proto:/Users/gaige/go/src/github.com/gogo/protobuf/protobuf:proto/github.com/argoproj/argo/pkg/apis/workflow/v1alpha1:.
WARN: ignoring type interface{}
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecretKeySelector" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ContainerPort" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvFromSource" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvVar" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ResourceRequirements" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeMount" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeDevice" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Lifecycle" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "TerminationMessagePolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "PullPolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecurityContext" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ContainerPort" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvFromSource" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "EnvVar" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "ResourceRequirements" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeMount" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "VolumeDevice" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Probe" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Lifecycle" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "TerminationMessagePolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "PullPolicy" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "SecurityContext" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Affinity" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Container" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Toleration" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "UID" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/types will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "OwnerReference" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Initializers" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Volume" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "PersistentVolumeClaim" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Affinity" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Toleration" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "LocalObjectReference" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Time" of package github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 will be ignored because it was not present on the scan path.
WARN: type "Volume" of package github.com/argoproj/argo/vendor/k8s.io/api/core/v1 will be ignored because it was not present on the scan path.
WARN: no RPCs in the given proto file, not generating anything
$ ls proto/
github.com
$ ls -lah  proto/github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto
-rwxr-xr-x  1 gaige  staff    12K May 22 12:29 proto/github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto

I am not really sure why proteus won't generate proto from your fork of argo. I suspect it could have to do with package names / paths. I had some issues when I was trying to generate proto in my own fork of argo before I opened this issue.

gaigepr commented 6 years ago

Another experiment this time using go-to-protobuf that doesn't quite succeed.

$ dep ensure -add github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
Fetching sources...
$ bash ./hack/generate-proto.sh
+ set -o errexit
+ set -o nounset
+ set -o pipefail
+++ dirname ./hack/generate-proto.sh
++ cd ./hack/..
++ pwd
+ PROJECT_ROOT=/Users/gaige/go/src/github.com/argoproj/argo
++ cd /Users/gaige/go/src/github.com/argoproj/argo
++ ls -d -1 ./vendor/k8s.io/code-generator
+ CODEGEN_PKG=./vendor/k8s.io/code-generator
+ PATH=/Users/gaige/go/src/github.com/argoproj/argo/dist:/Users/gaige/.cargo/bin:./node_modules/.bin:/Users/gaige/go/bin:/Users/gaige/.google/google-cloud-sdk/bin:/Users/gaige/.nvm/versions/node/v8.9.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
+ go build -i -o dist/go-to-protobuf ./vendor/k8s.io/code-generator/cmd/go-to-protobuf
+ go build -i -o dist/protoc-gen-gogo ./vendor/k8s.io/code-generator/cmd/go-to-protobuf/protoc-gen-gogo
+ PACKAGES=(github.com/argoproj/argo/pkg/apis/workflow/v1alpha1)
+ APIMACHINERY_PKGS=(+k8s.io/apimachinery/pkg/util/intstr +k8s.io/apimachinery/pkg/api/resource +k8s.io/apimachinery/pkg/runtime/schema +k8s.io/apimachinery/pkg/runtime k8s.io/apimachinery/pkg/apis/meta/v1 k8s.io/api/core/v1)
++ IFS=,
++ echo github.com/argoproj/argo/pkg/apis/workflow/v1alpha1
++ IFS=,
++ echo +k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1
+ go-to-protobuf --logtostderr --go-header-file=/Users/gaige/go/src/github.com/argoproj/argo/hack/custom-boilerplate.go.txt --packages=github.com/argoproj/argo/pkg/apis/workflow/v1alpha1 --apimachinery-packages=+k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1 --proto-import=./vendor
2018/05/22 14:02:23 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/api/resource/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:02:23 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/runtime/schema/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:02:23 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/runtime/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:02:23 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:02:24 github.com/argoproj/argo/vendor/k8s.io/api/core/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
2018/05/22 14:02:26 github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/generated.proto but not used.
+ go build -i -o dist/protoc-gen-gogofast ./vendor/github.com/gogo/protobuf/protoc-gen-gogofast
+ GOPROTOBINARY=gogofast
+ go build -i -o dist/protoc-gen-grpc-gateway ./vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
# github.com/argoproj/argo/vendor/google.golang.org/genproto/googleapis/api/annotations
vendor/google.golang.org/genproto/googleapis/api/annotations/http.pb.go:63:26: undefined: proto.InternalMessageInfo
vendor/google.golang.org/genproto/googleapis/api/annotations/http.pb.go:349:30: undefined: proto.InternalMessageInfo
vendor/google.golang.org/genproto/googleapis/api/annotations/http.pb.go:614:39: undefined: proto.InternalMessageInfo
gaigepr commented 6 years ago

Another go

$ pwd ; git status
$ pwd ; git status
/Users/gaige/go/src/github.com/argoproj/argo
On branch protobuf
Your branch is up to date with 'origin/protobuf'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   Gopkg.lock
    modified:   Gopkg.toml

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    hack/generate-proto.sh

no changes added to commit (use "git add" and/or "git commit -a")

Here are the changes made to the Gopkg.toml

$ git diff Gopkg.toml
diff --git a/Gopkg.toml b/Gopkg.toml
index 3976756..69484b1 100644
--- a/Gopkg.toml
+++ b/Gopkg.toml
@@ -5,6 +5,9 @@ required = [
   "k8s.io/code-generator/cmd/lister-gen",
   # required by packr build (which we don't import as a package)
   "golang.org/x/sync/errgroup",
+  # required for proto generation
+  "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway",
+  "github.com/golang/protobuf",
 ]

 [[constraint]]
@@ -50,3 +53,11 @@ required = [
 [[constraint]]
   name = "github.com/spf13/cobra"
   branch = "master"
+
+[[constraint]]
+  name = "github.com/grpc-ecosystem/grpc-gateway"
+  version = "1.4.0"
+
+[[constraint]]
+  name = "github.com/gogo/protobuf"
+  version = "0.5.0"

Now, lets try generating stuff again! I have made sure protoc, protoc-gen-go, et al. are updated.

$ bash ./hack/generate-proto.sh
+ set -o errexit
+ set -o nounset
+ set -o pipefail
+++ dirname ./hack/generate-proto.sh
++ cd ./hack/..
++ pwd
+ PROJECT_ROOT=/Users/gaige/go/src/github.com/argoproj/argo
++ cd /Users/gaige/go/src/github.com/argoproj/argo
++ ls -d -1 ./vendor/k8s.io/code-generator
+ CODEGEN_PKG=./vendor/k8s.io/code-generator
+ PATH=/Users/gaige/go/src/github.com/argoproj/argo/dist:/Users/gaige/.cargo/bin:./node_modules/.bin:/Users/gaige/go/bin:/Users/gaige/.google/google-cloud-sdk/bin:/Users/gaige/.nvm/versions/node/v8.9.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
+ go build -i -o dist/go-to-protobuf ./vendor/k8s.io/code-generator/cmd/go-to-protobuf
+ go build -i -o dist/protoc-gen-gogo ./vendor/k8s.io/code-generator/cmd/go-to-protobuf/protoc-gen-gogo
+ PACKAGES=(github.com/argoproj/argo/pkg/apis/workflow/v1alpha1)
+ APIMACHINERY_PKGS=(+k8s.io/apimachinery/pkg/util/intstr +k8s.io/apimachinery/pkg/api/resource +k8s.io/apimachinery/pkg/runtime/schema +k8s.io/apimachinery/pkg/runtime k8s.io/apimachinery/pkg/apis/meta/v1 k8s.io/api/core/v1)
++ IFS=,
++ echo github.com/argoproj/argo/pkg/apis/workflow/v1alpha1
++ IFS=,
++ echo +k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1
+ go-to-protobuf --logtostderr --go-header-file=/Users/gaige/go/src/github.com/argoproj/argo/hack/custom-boilerplate.go.txt --packages=github.com/argoproj/argo/pkg/apis/workflow/v1alpha1 --apimachinery-packages=+k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1 --proto-import=./vendor
2018/05/22 14:32:41 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/api/resource/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:32:41 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/runtime/schema/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:32:41 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/runtime/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:32:41 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 14:32:42 github.com/argoproj/argo/vendor/k8s.io/api/core/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
2018/05/22 14:32:44 github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/generated.proto but not used.
+ go build -i -o dist/protoc-gen-gogofast ./vendor/github.com/gogo/protobuf/protoc-gen-gogofast
+ GOPROTOBINARY=gogofast
+ go build -i -o dist/protoc-gen-grpc-gateway ./vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
# github.com/argoproj/argo/vendor/google.golang.org/genproto/googleapis/api/annotations
vendor/google.golang.org/genproto/googleapis/api/annotations/http.pb.go:63:26: undefined: proto.InternalMessageInfo
vendor/google.golang.org/genproto/googleapis/api/annotations/http.pb.go:349:30: undefined: proto.InternalMessageInfo
vendor/google.golang.org/genproto/googleapis/api/annotations/http.pb.go:614:39: undefined: proto.InternalMessageInfo

It seems to exit as normal because of some undefined proto messages... But look!

$ git status
On branch protobuf
Your branch is up to date with 'origin/protobuf'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   Gopkg.lock
    modified:   Gopkg.toml
    modified:   pkg/apis/workflow/v1alpha1/types.go

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    hack/generate-proto.sh
    pkg/apis/workflow/v1alpha1/generated.pb.go
    pkg/apis/workflow/v1alpha1/generated.proto

no changes added to commit (use "git add" and/or "git commit -a")

So let's look at what changed.

$ git diff pkg/apis/workflow/v1alpha1/types.go
diff --git a/pkg/apis/workflow/v1alpha1/types.go b/pkg/apis/workflow/v1alpha1/types.go
index c85d824..29c7238 100644
--- a/pkg/apis/workflow/v1alpha1/types.go
+++ b/pkg/apis/workflow/v1alpha1/types.go
@@ -53,215 +53,215 @@ const (
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
 type Workflow struct {
        metav1.TypeMeta   `json:",inline"`
-       metav1.ObjectMeta `json:"metadata"`
-       Spec              WorkflowSpec   `json:"spec"`
-       Status            WorkflowStatus `json:"status"`
+       metav1.ObjectMeta `json:"metadata" protobuf:"bytes,1,opt,name=metadata"`
+       Spec              WorkflowSpec   `json:"spec" protobuf:"bytes,2,opt,name=spec"`
+       Status            WorkflowStatus `json:"status" protobuf:"bytes,3,opt,name=status"`
 }

 // WorkflowList is list of Workflow resources
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
 type WorkflowList struct {
        metav1.TypeMeta `json:",inline"`
-       metav1.ListMeta `json:"metadata"`
-       Items           []Workflow `json:"items"`
+       metav1.ListMeta `json:"metadata" protobuf:"bytes,1,opt,name=metadata"`
+       Items           []Workflow `json:"items" protobuf:"bytes,2,rep,name=items"`
 }

 // WorkflowSpec is the specification of a Workflow.
 type WorkflowSpec struct {
        // Templates is a list of workflow templates used in a workflow
-       Templates []Template `json:"templates"`
+       Templates []Template `json:"templates" protobuf:"bytes,1,rep,name=templates"`

        // Entrypoint is a template reference to the starting point of the workflow
-       Entrypoint string `json:"entrypoint"`
+       Entrypoint string `json:"entrypoint" protobuf:"bytes,2,opt,name=entrypoint"`

        // Arguments contain the parameters and artifacts sent to the workflow entrypoint
        // Parameters are referencable globally using the 'workflow' variable prefix.
        // e.g. {{workflow.parameters.myparam}}
-       Arguments Arguments `json:"arguments,omitempty"`
+       Arguments Arguments `json:"arguments,omitempty" protobuf:"bytes,3,opt,name=arguments"`

        // ServiceAccountName is the name of the ServiceAccount to run all pods of the workflow as.
-       ServiceAccountName string `json:"serviceAccountName,omitempty"`
+       ServiceAccountName string `json:"serviceAccountName,omitempty" protobuf:"bytes,4,opt,name=serviceAccountName"`

        // Volumes is a list of volumes that can be mounted by containers in a workflow.
-       Volumes []apiv1.Volume `json:"volumes,omitempty"`
+       Volumes []apiv1.Volume `json:"volumes,omitempty" protobuf:"bytes,5,rep,name=volumes"`

        // VolumeClaimTemplates is a list of claims that containers are allowed to reference.
        // The Workflow controller will create the claims at the beginning of the workflow
        // and delete the claims upon completion of the workflow
-       VolumeClaimTemplates []apiv1.PersistentVolumeClaim `json:"volumeClaimTemplates,omitempty"`
+       VolumeClaimTemplates []apiv1.PersistentVolumeClaim `json:"volumeClaimTemplates,omitempty" protobuf:"bytes,6,rep,name=volumeClaimTemplates"`

        // Parallelism limits the max total parallel pods that can execute at the same time in a workflow
-       Parallelism *int64 `json:"parallelism,omitempty"`
+       Parallelism *int64 `json:"parallelism,omitempty" protobuf:"varint,7,opt,name=parallelism"`

        // Suspend will suspend the workflow and prevent execution of any future steps in the workflow
-       Suspend *bool `json:"suspend,omitempty"`
+       Suspend *bool `json:"suspend,omitempty" protobuf:"varint,8,opt,name=suspend"`

        // NodeSelector is a selector which will result in all pods of the workflow
        // to be scheduled on the selected node(s). This is able to be overridden by
        // a nodeSelector specified in the template.
-       NodeSelector map[string]string `json:"nodeSelector,omitempty"`
+       NodeSelector map[string]string `json:"nodeSelector,omitempty" protobuf:"bytes,9,rep,name=nodeSelector"`

        // Affinity sets the scheduling constraints for all pods in the workflow.
        // Can be overridden by an affinity specified in the template
-       Affinity *apiv1.Affinity `json:"affinity,omitempty"`
+       Affinity *apiv1.Affinity `json:"affinity,omitempty" protobuf:"bytes,10,opt,name=affinity"`

        // Tolerations to apply to workflow pods.
-       Tolerations []apiv1.Toleration `json:"tolerations,omitempty"`
+       Tolerations []apiv1.Toleration `json:"tolerations,omitempty" protobuf:"bytes,11,rep,name=tolerations"`

        // ImagePullSecrets is a list of references to secrets in the same namespace to use for pulling any images
        // in pods that reference this ServiceAccount. ImagePullSecrets are distinct from Secrets because Secrets
        // can be mounted in the pod, but ImagePullSecrets are only accessed by the kubelet.
        // More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
-       ImagePullSecrets []apiv1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
+       ImagePullSecrets []apiv1.LocalObjectReference `json:"imagePullSecrets,omitempty" protobuf:"bytes,12,rep,name=imagePullSecrets"`

        // OnExit is a template reference which is invoked at the end of the
        // workflow, irrespective of the success, failure, or error of the
        // primary workflow.
-       OnExit string `json:"onExit,omitempty"`
+       OnExit string `json:"onExit,omitempty" protobuf:"bytes,13,opt,name=onExit"`
 }
# ... The rest is omitted for brevity but you get the idea

So that is interesting, it annotated our go structs! Now lets look at those other files that are new.

$ cat pkg/apis/workflow/v1alpha1/generated.proto

// This file was autogenerated by go-to-protobuf. Do not edit it manually!

syntax = 'proto2';

package github.com.argoproj.argo.pkg.apis.workflow.v1alpha1;

import "k8s.io/api/core/v1/generated.proto";
import "k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto";
import "k8s.io/apimachinery/pkg/runtime/generated.proto";
import "k8s.io/apimachinery/pkg/runtime/schema/generated.proto";
import "k8s.io/apimachinery/pkg/util/intstr/generated.proto";

// Package-wide variables from generator "generated".
option go_package = "v1alpha1";

// Arguments to a template
message Arguments {
  // Parameters is the list of parameters to pass to the template or workflow
  repeated Parameter parameters = 1;

  // Artifacts is the list of artifacts to pass to the template or workflow
  repeated Artifact artifacts = 2;
}

// Artifact indicates an artifact to place at a specified path
message Artifact {
  // name of the artifact. must be unique within a template's inputs/outputs.
  optional string name = 1;

  // Path is the container path to the artifact
  optional string path = 2;

  // mode bits to use on this file, must be a value between 0 and 0777
  // set when loading input artifacts.
  optional int32 mode = 3;

  // From allows an artifact to reference an artifact from a previous step
  optional string from = 4;

  // ArtifactLocation contains the location of the artifact
  optional ArtifactLocation artifactLocation = 5;

  // GlobalName exports an output artifact to the global scope, making it available as
  // '{{workflow.outputs.artifacts.XXXX}} and in workflow.status.outputs.artifacts
  optional string globalName = 6;
}

// ArtifactLocation describes a location for a single or multiple artifacts.
// It is used as single artifact in the context of inputs/outputs (e.g. outputs.artifacts.artname).
// It is also used to describe the location of multiple artifacts such as the archive location
// of a single workflow step, which the executor will use as a default location to store its files.
message ArtifactLocation {
  // S3 contains S3 artifact location details
  optional S3Artifact s3 = 1;

  // Git contains git artifact location details
  optional GitArtifact git = 2;

  // HTTP contains HTTP artifact location details
  optional HTTPArtifact http = 3;

  // Artifactory contains artifactory artifact location details
  optional ArtifactoryArtifact artifactory = 4;

  // Raw contains raw artifact location details
  optional RawArtifact raw = 5;
}

// ArtifactoryArtifact is the location of an artifactory artifact
message ArtifactoryArtifact {
  // URL of the artifact
  optional string url = 1;

  optional ArtifactoryAuth artifactoryAuth = 2;
}

// ArtifactoryAuth describes the secret selectors required for authenticating to artifactory
message ArtifactoryAuth {
  // UsernameSecret is the secret selector to the repository username
  optional k8s.io.api.core.v1.SecretKeySelector usernameSecret = 1;

  // PasswordSecret is the secret selector to the repository password
  optional k8s.io.api.core.v1.SecretKeySelector passwordSecret = 2;
}

// DAGTask represents a node in the graph during DAG execution
message DAGTask {
  // Name is the name of the target
  optional string name = 1;

  // Name of template to execute
  optional string template = 2;

  // Arguments are the parameter and artifact arguments to the template
  optional Arguments arguments = 3;

  // Dependencies are name of other targets which this depends on
  repeated string dependencies = 4;
}

// DAGTemplate is a template subtype for directed acyclic graph templates
message DAGTemplate {
  // Target are one or more names of targets to execute in a DAG
  optional string target = 1;

  // Tasks are a list of DAG tasks
  repeated DAGTask tasks = 2;
}

// GitArtifact is the location of an git artifact
message GitArtifact {
  // Repo is the git repository
  optional string repo = 1;

  // Revision is the git commit, tag, branch to checkout
  optional string revision = 2;

  // UsernameSecret is the secret selector to the repository username
  optional k8s.io.api.core.v1.SecretKeySelector usernameSecret = 3;

  // PasswordSecret is the secret selector to the repository password
  optional k8s.io.api.core.v1.SecretKeySelector passwordSecret = 4;
}

// HTTPArtifact allows an file served on HTTP to be placed as an input artifact in a container
message HTTPArtifact {
  // URL of the artifact
  optional string url = 1;
}

// Inputs are the mechanism for passing parameters, artifacts, volumes from one template to another
message Inputs {
  // Parameters are a list of parameters passed as inputs
  repeated Parameter parameters = 1;

  // Artifact are a list of artifacts passed as inputs
  repeated Artifact artifacts = 2;
}

// Item expands a single workflow step into multiple parallel steps
// The value of Item can be a map, string, bool, or number
//
// +protobuf=true
// +protobuf.options.(gogoproto.goproto_stringer)=false
// +k8s:openapi-gen=true
message Item {
  optional int64 type = 1;

  optional string numVal = 2;

  optional bool boolVal = 3;

  optional string strVal = 4;

  map<string, ItemValue> mapVal = 5;
}

// +protobuf=true
// +protobuf.options.(gogoproto.goproto_stringer)=false
// +k8s:openapi-gen=true
message ItemValue {
  optional int64 type = 1;

  optional string numVal = 2;

  optional bool boolVal = 3;

  optional string strVal = 4;
}

// Pod metdata
message Metadata {
  map<string, string> annotations = 1;

  map<string, string> labels = 2;
}

// NodeStatus contains status information about an individual node in the workflow
// +k8s:openapi-gen=false
message NodeStatus {
  // ID is a unique identifier of a node within the worklow
  // It is implemented as a hash of the node name, which makes the ID deterministic
  optional string id = 1;

  // Name is unique name in the node tree used to generate the node ID
  optional string name = 2;

  // DisplayName is a human readable representation of the node. Unique within a template boundary
  optional string displayName = 3;

  // Type indicates type of node
  optional string type = 4;

  // TemplateName is the template name which this node corresponds to. Not applicable to virtual nodes (e.g. Retry, StepGroup)
  optional string templateName = 5;

  // Phase a simple, high-level summary of where the node is in its lifecycle.
  // Can be used as a state machine.
  optional string phase = 6;

  // BoundaryID indicates the node ID of the associated template root node in which this node belongs to
  optional string boundaryID = 7;

  // A human readable message indicating details about why the node is in this condition.
  optional string message = 8;

  // Time at which this node started
  optional k8s.io.apimachinery.pkg.apis.meta.v1.Time startedAt = 9;

  // Time at which this node completed
  optional k8s.io.apimachinery.pkg.apis.meta.v1.Time finishedAt = 10;

  // PodIP captures the IP of the pod for daemoned steps
  optional string podIP = 11;

  // Daemoned tracks whether or not this node was daemoned and need to be terminated
  optional bool daemoned = 12;

  // Inputs captures input parameter values and artifact locations supplied to this template invocation
  optional Inputs inputs = 13;

  // Outputs captures output parameter values and artifact locations produced by this template invocation
  optional Outputs outputs = 14;

  // Children is a list of child node IDs
  repeated string children = 15;

  // OutboundNodes tracks the node IDs which are considered "outbound" nodes to a template invocation.
  // For every invocation of a template, there are nodes which we considered as "outbound". Essentially,
  // these are last nodes in the execution sequence to run, before the template is considered completed.
  // These nodes are then connected as parents to a following step.
  //
  // In the case of single pod steps (i.e. container, script, resource templates), this list will be nil
  // since the pod itself is already considered the "outbound" node.
  // In the case of DAGs, outbound nodes are the "target" tasks (tasks with no children).
  // In the case of steps, outbound nodes are all the containers involved in the last step group.
  // NOTE: since templates are composable, the list of outbound nodes are carried upwards when
  // a DAG/steps template invokes another DAG/steps template. In other words, the outbound nodes of
  // a template, will be a superset of the outbound nodes of its last children.
  repeated string outboundNodes = 16;
}

// Outputs hold parameters, artifacts, and results from a step
message Outputs {
  // Parameters holds the list of output parameters produced by a step
  repeated Parameter parameters = 1;

  // Artifacts holds the list of output artifacts produced by a step
  repeated Artifact artifacts = 2;

  // Result holds the result (stdout) of a script template
  optional string result = 3;
}

// Parameter indicate a passed string parameter to a service template with an optional default value
message Parameter {
  // Name is the parameter name
  optional string name = 1;

  // Default is the default value to use for an input parameter if a value was not supplied
  optional string default = 2;

  // Value is the literal value to use for the parameter.
  // If specified in the context of an input parameter, the value takes precedence over any passed values
  optional string value = 3;

  // ValueFrom is the source for the output parameter's value
  optional ValueFrom valueFrom = 4;

  // GlobalName exports an output parameter to the global scope, making it available as
  // '{{workflow.outputs.parameters.XXXX}} and in workflow.status.outputs.parameters
  optional string globalName = 5;
}

// RawArtifact allows raw string content to be placed as an artifact in a container
message RawArtifact {
  // Data is the string contents of the artifact
  optional string data = 1;
}

// ResourceTemplate is a template subtype to manipulate kubernetes resources
message ResourceTemplate {
  // Action is the action to perform to the resource.
  // Must be one of: get, create, apply, delete, replace
  optional string action = 1;

  // Manifest contains the kubernetes manifest
  optional string manifest = 2;

  // SuccessCondition is a label selector expression which describes the conditions
  // of the k8s resource in which it is acceptable to proceed to the following step
  optional string successCondition = 3;

  // FailureCondition is a label selector expression which describes the conditions
  // of the k8s resource in which the step was considered failed
  optional string failureCondition = 4;
}

// RetryStrategy provides controls on how to retry a workflow step
message RetryStrategy {
  // Limit is the maximum number of attempts when retrying a container
  optional int32 limit = 1;
}

// S3Artifact is the location of an S3 artifact
message S3Artifact {
  optional S3Bucket s3Bucket = 1;

  // Key is the key in the bucket where the artifact resides
  optional string key = 2;
}

// S3Bucket contains the access information required for interfacing with an S3 bucket
message S3Bucket {
  // Endpoint is the hostname of the bucket endpoint
  optional string endpoint = 1;

  // Bucket is the name of the bucket
  optional string bucket = 2;

  // Region contains the optional bucket region
  optional string region = 3;

  // Insecure will connect to the service with TLS
  optional bool insecure = 4;

  // AccessKeySecret is the secret selector to the bucket's access key
  optional k8s.io.api.core.v1.SecretKeySelector accessKeySecret = 5;

  // SecretKeySecret is the secret selector to the bucket's secret key
  optional k8s.io.api.core.v1.SecretKeySelector secretKeySecret = 6;
}

// ScriptTemplate is a template subtype to enable scripting through code steps
message ScriptTemplate {
  optional k8s.io.api.core.v1.Container container = 1;

  // Source contains the source code of the script to execute
  optional string source = 2;
}

// Sidecar is a container which runs alongside the main container
message Sidecar {
  optional k8s.io.api.core.v1.Container container = 1;

  // MirrorVolumeMounts will mount the same volumes specified in the main container
  // to the sidecar (including artifacts), at the same mountPaths. This enables
  // dind daemon to partially see the same filesystem as the main container in
  // order to use features such as docker volume binding
  optional bool mirrorVolumeMounts = 2;
}

// SuspendTemplate is a template subtype to suspend a workflow at a predetermined point in time
message SuspendTemplate {
}

// Template is a reusable and composable unit of execution in a workflow
message Template {
  // Name is the name of the template
  optional string name = 1;

  // Inputs describe what inputs parameters and artifacts are supplied to this template
  optional Inputs inputs = 2;

  // Outputs describe the parameters and artifacts that this template produces
  optional Outputs outputs = 3;

  // NodeSelector is a selector to schedule this step of the workflow to be
  // run on the selected node(s). Overrides the selector set at the workflow level.
  map<string, string> nodeSelector = 4;

  // Affinity sets the pod's scheduling constraints
  // Overrides the affinity set at the workflow level (if any)
  optional k8s.io.api.core.v1.Affinity affinity = 5;

  // Metdata sets the pods's metadata, i.e. annotations and labels
  optional Metadata metadata = 6;

  // Deamon will allow a workflow to proceed to the next step so long as the container reaches readiness
  optional bool daemon = 7;

  // Steps define a series of sequential/parallel workflow steps
  repeated WorkflowStep steps = 8;

  // Container is the main container image to run in the pod
  optional k8s.io.api.core.v1.Container container = 9;

  // Script runs a portion of code against an interpreter
  optional ScriptTemplate script = 10;

  // Resource template subtype which can run k8s resources
  optional ResourceTemplate resource = 11;

  // DAG template subtype which runs a DAG
  optional DAGTemplate dag = 12;

  // Suspend template subtype which can suspend a workflow when reaching the step
  optional SuspendTemplate suspend = 13;

  // Sidecars is a list of containers which run alongside the main container
  // Sidecars are automatically killed when the main container completes
  repeated Sidecar sidecars = 14;

  // Location in which all files related to the step will be stored (logs, artifacts, etc...).
  // Can be overridden by individual items in Outputs. If omitted, will use the default
  // artifact repository location configured in the controller, appended with the
  // <workflowname>/<nodename> in the key.
  optional ArtifactLocation archiveLocation = 15;

  // Optional duration in seconds relative to the StartTime that the pod may be active on a node
  // before the system actively tries to terminate the pod; value must be positive integer
  // This field is only applicable to container and script templates.
  optional int64 activeDeadlineSeconds = 16;

  // RetryStrategy describes how to retry a template when it fails
  optional RetryStrategy retryStrategy = 17;

  // Parallelism limits the max total parallel pods that can execute at the same time within the
  // boundaries of this template invocation. If additional steps/dag templates are invoked, the
  // pods created by those templates will not be counted towards this total.
  optional int64 parallelism = 18;

  // Tolerations to apply to workflow pods.
  repeated k8s.io.api.core.v1.Toleration tolerations = 19;
}

// ValueFrom describes a location in which to obtain the value to a parameter
message ValueFrom {
  // Path in the container to retrieve an output parameter value from in container templates
  optional string path = 1;

  // JSONPath of a resource to retrieve an output parameter value from in resource templates
  optional string jsonPath = 2;

  // JQFilter expression against the resource object in resource templates
  optional string jqFilter = 3;

  // Parameter reference to a step or dag task in which to retrieve an output parameter value from
  // (e.g. '{{steps.mystep.outputs.myparam}}')
  optional string parameter = 4;
}

// Workflow is the definition of a workflow resource
// +genclient
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
message Workflow {
  optional k8s.io.apimachinery.pkg.apis.meta.v1.ObjectMeta metadata = 1;

  optional WorkflowSpec spec = 2;

  optional WorkflowStatus status = 3;
}

// WorkflowList is list of Workflow resources
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
message WorkflowList {
  optional k8s.io.apimachinery.pkg.apis.meta.v1.ListMeta metadata = 1;

  repeated Workflow items = 2;
}

// WorkflowSpec is the specification of a Workflow.
message WorkflowSpec {
  // Templates is a list of workflow templates used in a workflow
  repeated Template templates = 1;

  // Entrypoint is a template reference to the starting point of the workflow
  optional string entrypoint = 2;

  // Arguments contain the parameters and artifacts sent to the workflow entrypoint
  // Parameters are referencable globally using the 'workflow' variable prefix.
  // e.g. {{workflow.parameters.myparam}}
  optional Arguments arguments = 3;

  // ServiceAccountName is the name of the ServiceAccount to run all pods of the workflow as.
  optional string serviceAccountName = 4;

  // Volumes is a list of volumes that can be mounted by containers in a workflow.
  repeated k8s.io.api.core.v1.Volume volumes = 5;

  // VolumeClaimTemplates is a list of claims that containers are allowed to reference.
  // The Workflow controller will create the claims at the beginning of the workflow
  // and delete the claims upon completion of the workflow
  repeated k8s.io.api.core.v1.PersistentVolumeClaim volumeClaimTemplates = 6;

  // Parallelism limits the max total parallel pods that can execute at the same time in a workflow
  optional int64 parallelism = 7;

  // Suspend will suspend the workflow and prevent execution of any future steps in the workflow
  optional bool suspend = 8;

  // NodeSelector is a selector which will result in all pods of the workflow
  // to be scheduled on the selected node(s). This is able to be overridden by
  // a nodeSelector specified in the template.
  map<string, string> nodeSelector = 9;

  // Affinity sets the scheduling constraints for all pods in the workflow.
  // Can be overridden by an affinity specified in the template
  optional k8s.io.api.core.v1.Affinity affinity = 10;

  // Tolerations to apply to workflow pods.
  repeated k8s.io.api.core.v1.Toleration tolerations = 11;

  // ImagePullSecrets is a list of references to secrets in the same namespace to use for pulling any images
  // in pods that reference this ServiceAccount. ImagePullSecrets are distinct from Secrets because Secrets
  // can be mounted in the pod, but ImagePullSecrets are only accessed by the kubelet.
  // More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  repeated k8s.io.api.core.v1.LocalObjectReference imagePullSecrets = 12;

  // OnExit is a template reference which is invoked at the end of the
  // workflow, irrespective of the success, failure, or error of the
  // primary workflow.
  optional string onExit = 13;
}

// WorkflowStatus contains overall status information about a workflow
// +k8s:openapi-gen=false
message WorkflowStatus {
  // Phase a simple, high-level summary of where the workflow is in its lifecycle.
  optional string phase = 1;

  // Time at which this workflow started
  optional k8s.io.apimachinery.pkg.apis.meta.v1.Time startedAt = 2;

  // Time at which this workflow completed
  optional k8s.io.apimachinery.pkg.apis.meta.v1.Time finishedAt = 3;

  // A human readable message indicating details about why the workflow is in this condition.
  optional string message = 4;

  // Nodes is a mapping between a node ID and the node's status.
  map<string, NodeStatus> nodes = 5;

  // PersistentVolumeClaims tracks all PVCs that were created as part of the workflow.
  // The contents of this list are drained at the end of the workflow.
  repeated k8s.io.api.core.v1.Volume persistentVolumeClaims = 6;

  // Outputs captures output values and artifact locations produced by the workflow via global outputs
  optional Outputs outputs = 7;
}

// WorkflowStep is a reference to a template to execute in a series of step
message WorkflowStep {
  // Name of the step
  optional string name = 1;

  // Template is a reference to the template to execute as the step
  optional string template = 2;

  // Arguments hold arguments to the template
  optional Arguments arguments = 3;

  // WithItems expands a step into multiple parallel steps from the items in the list
  repeated Item withItems = 4;

  // WithParam expands a step into from the value in the parameter
  optional string withParam = 5;

  // When is an expression in which the step should conditionally execute
  optional string when = 6;
}
gaigepr commented 6 years ago

So, partial progress but the bash script doesn't run to completion so I cannot attest to the quality of the proto.

gaigepr commented 6 years ago

Another experiment. This one removes references to grpc-gateway from the hack/generate-proto.sh script.

$ bash hack/generate-proto.sh
+ set -o errexit
+ set -o nounset
+ set -o pipefail
+++ dirname hack/generate-proto.sh
++ cd hack/..
++ pwd
+ PROJECT_ROOT=/Users/gaige/go/src/github.com/argoproj/argo
++ cd /Users/gaige/go/src/github.com/argoproj/argo
++ ls -d -1 ./vendor/k8s.io/code-generator
+ CODEGEN_PKG=./vendor/k8s.io/code-generator
+ PATH=/Users/gaige/go/src/github.com/argoproj/argo/dist:/Users/gaige/.cargo/bin:./node_modules/.bin:/Users/gaige/go/bin:/Users/gaige/.google/google-cloud-sdk/bin:/Users/gaige/.nvm/versions/node/v8.9.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
+ go build -i -o dist/go-to-protobuf ./vendor/k8s.io/code-generator/cmd/go-to-protobuf
+ go build -i -o dist/protoc-gen-gogo ./vendor/k8s.io/code-generator/cmd/go-to-protobuf/protoc-gen-gogo
+ PACKAGES=(github.com/argoproj/argo/pkg/apis/workflow/v1alpha1)
+ APIMACHINERY_PKGS=(+k8s.io/apimachinery/pkg/util/intstr +k8s.io/apimachinery/pkg/api/resource +k8s.io/apimachinery/pkg/runtime/schema +k8s.io/apimachinery/pkg/runtime k8s.io/apimachinery/pkg/apis/meta/v1 k8s.io/api/core/v1)
++ IFS=,
++ echo github.com/argoproj/argo/pkg/apis/workflow/v1alpha1
++ IFS=,
++ echo +k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1
+ go-to-protobuf --logtostderr --go-header-file=/Users/gaige/go/src/github.com/argoproj/argo/hack/custom-boilerplate.go.txt --packages=github.com/argoproj/argo/pkg/apis/workflow/v1alpha1 --apimachinery-packages=+k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1 --proto-import=./vendor
2018/05/22 16:25:19 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/api/resource/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 16:25:19 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/runtime/schema/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 16:25:20 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/runtime/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 16:25:20 github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
github.com/argoproj/argo/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
2018/05/22 16:25:21 github.com/argoproj/argo/vendor/k8s.io/api/core/v1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
2018/05/22 16:25:23 github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/util/intstr/generated.proto but not used.
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/schema/generated.proto but not used.
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/generated.proto: warning: Import k8s.io/apimachinery/pkg/runtime/generated.proto but not used.
+ go build -i -o dist/protoc-gen-gogofast ./vendor/github.com/gogo/protobuf/protoc-gen-gogofast
+ GOPROTOBINARY=gogofast
++ find /Users/gaige/go/src/github.com/argoproj/argo '(' -name '*.proto' -and -path '*/server/*' -or -path '*/reposerver/*' -and -name '*.proto' ')'
+ PROTO_FILES='/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/reposerver/repository/repository.proto
/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/application/application.proto
/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/cluster/cluster.proto
/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/repository/repository.proto
/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/version/version.proto'
+ for i in '${PROTO_FILES}'
+ '[' -f /.dockerenv ']'
+ GOOGLE_PROTO_API_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
+ GOGO_PROTOBUF_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf
+ protoc -I/Users/gaige/go/src/github.com/argoproj/argo -I/usr/local/include -I./vendor -I/Users/gaige/go/src -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf --gogofast_out=plugins=grpc:/Users/gaige/go/src /Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/reposerver/repository/repository.proto
vendor/github.com/argoproj/argo-cd/reposerver/repository/repository.proto: warning: Import k8s.io/api/core/v1/generated.proto but not used.
+ for i in '${PROTO_FILES}'
+ '[' -f /.dockerenv ']'
+ GOOGLE_PROTO_API_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
+ GOGO_PROTOBUF_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf
+ protoc -I/Users/gaige/go/src/github.com/argoproj/argo -I/usr/local/include -I./vendor -I/Users/gaige/go/src -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf --gogofast_out=plugins=grpc:/Users/gaige/go/src /Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/application/application.proto
+ for i in '${PROTO_FILES}'
+ '[' -f /.dockerenv ']'
+ GOOGLE_PROTO_API_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
+ GOGO_PROTOBUF_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf
+ protoc -I/Users/gaige/go/src/github.com/argoproj/argo -I/usr/local/include -I./vendor -I/Users/gaige/go/src -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf --gogofast_out=plugins=grpc:/Users/gaige/go/src /Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/cluster/cluster.proto
+ for i in '${PROTO_FILES}'
+ '[' -f /.dockerenv ']'
+ GOOGLE_PROTO_API_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
+ GOGO_PROTOBUF_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf
+ protoc -I/Users/gaige/go/src/github.com/argoproj/argo -I/usr/local/include -I./vendor -I/Users/gaige/go/src -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf --gogofast_out=plugins=grpc:/Users/gaige/go/src /Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/repository/repository.proto
vendor/github.com/argoproj/argo-cd/server/repository/repository.proto: warning: Import k8s.io/api/core/v1/generated.proto but not used.
+ for i in '${PROTO_FILES}'
+ '[' -f /.dockerenv ']'
+ GOOGLE_PROTO_API_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
+ GOGO_PROTOBUF_PATH=/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf
+ protoc -I/Users/gaige/go/src/github.com/argoproj/argo -I/usr/local/include -I./vendor -I/Users/gaige/go/src -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I/Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/gogo/protobuf --gogofast_out=plugins=grpc:/Users/gaige/go/src /Users/gaige/go/src/github.com/argoproj/argo/vendor/github.com/argoproj/argo-cd/server/version/version.proto

YAHOO! The script runs to completion and does not complain about anything being undefined.