knative / serving

Kubernetes-based, scale-to-zero, request-driven compute
https://knative.dev/docs/serving/
Apache License 2.0
5.56k stars 1.16k forks source link

hack/update-codegen.sh is broken on macOS 10.14.6 (Mojave) #9792

Closed jsoref closed 3 years ago

jsoref commented 4 years ago

In what area(s)?

/area build

What version of Knative?

master?

$ git describe --dirty
fatal: No names found, cannot describe anything.

Expected Behavior

running:

./hack/update-deps.sh

should work (I don't have a definition for this, as so far, I have not managed to see this)

Actual Behavior

Initially https://github.com/knative/serving/pull/9778#issuecomment-707473327: $ hack/update-codegen.sh hack/../vendor/knative.dev/test-infra/scripts/library.sh: line 25: conditional binary operator expected

$ hack/update-codegen.sh
Generating checksums for configmap _example keys
Generating deepcopy funcs
Generating clientset for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/clientset
Generating listers for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/listers
Generating informers for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/informers
Generating injection for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/injection
=== Update Deps for Golang
--- Go mod tidy and vendor
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/networking/nscert/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/dataplane-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/deployment-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/load-test/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/rollout-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/scale-from-zero/continuous/kodata/refs
go: downloading knative.dev/networking v0.0.0-20201009180721-8cc9a61966f9
go: downloading knative.dev/pkg v0.0.0-20201009175121-3c4df8c27293
go: downloading golang.org/x/net v0.0.0-20200904194848-62affa334b73
go: downloading k8s.io/code-generator v0.18.8
go: downloading go.opencensus.io v0.22.4
go: downloading github.com/prometheus/client_golang v0.9.2
go: downloading github.com/google/go-cmp v0.5.2
go: downloading knative.dev/caching v0.0.0-20201009023721-0a00448bbad0
go: downloading github.com/google/go-containerregistry v0.1.3
go: downloading github.com/google/mako v0.0.0-20190821191249-122f8dcef9e3
go: downloading knative.dev/test-infra v0.0.0-20201009204121-322fb08edae7
go: downloading contrib.go.opencensus.io/exporter/stackdriver v0.13.2
go: downloading google.golang.org/grpc v1.31.1
go: downloading k8s.io/gengo v0.0.0-20200205140755-e0e292d8aa12
go: downloading golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.2.1-0.20200609204449-6bcf6f8577f0
go: downloading github.com/kelseyhightower/envconfig v1.4.0
go: downloading github.com/googleapis/gnostic v0.4.0
go: downloading go.uber.org/zap v1.15.0
go: downloading google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d
go: downloading golang.org/x/text v0.3.3
go: downloading github.com/openzipkin/zipkin-go v0.2.2
go: downloading github.com/mattbaird/jsonpatch v0.0.0-20171005235357-81af80346b1a
go: downloading github.com/gorilla/websocket v1.4.2
go: downloading gopkg.in/yaml.v3 v3.0.0-20190709130402-674ba3eaed22
go: downloading google.golang.org/protobuf v1.25.0
go: downloading github.com/vdemeester/k8s-pkg-credentialprovider v1.17.4
go: downloading github.com/prometheus/statsd_exporter v0.15.0
go: downloading sigs.k8s.io/structured-merge-diff v1.0.1-0.20191108220359-b1b620dd3f06
go: downloading cloud.google.com/go v0.65.0
go: downloading sigs.k8s.io/structured-merge-diff/v3 v3.0.1-0.20200706213357-43c19bbb7fba
go: downloading github.com/aws/aws-sdk-go v1.31.12
go: downloading github.com/census-instrumentation/opencensus-proto v0.3.0
go: downloading github.com/json-iterator/go v1.1.10
go: downloading gomodules.xyz/jsonpatch/v2 v2.1.0
go: downloading google.golang.org/appengine v1.6.6
go: downloading github.com/tsenart/vegeta v1.2.1-0.20190917092155-ab06ddb56e2f
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading google.golang.org/api v0.31.0
go: downloading github.com/imdario/mergo v0.3.9
go: downloading github.com/go-logr/logr v0.1.0
go: downloading github.com/influxdata/tdigest v0.0.1
go: downloading golang.org/x/tools v0.0.0-20200916195026-c9a70fc28ce3
go: downloading k8s.io/apiextensions-apiserver v0.18.8
go: downloading github.com/gorilla/mux v1.7.4
go: downloading go.uber.org/automaxprocs v1.3.0
go: downloading k8s.io/utils v0.0.0-20200603063816-c1c6865ac451
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading github.com/googleapis/gax-go v2.0.2+incompatible
go: downloading golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f
go: downloading honnef.co/go/tools v0.0.1-2020.1.5
go: downloading github.com/mailru/easyjson v0.7.1-0.20191009090205-6c0755d89d1e
go: downloading github.com/prometheus/procfs v0.0.11
go: downloading github.com/jmespath/go-jmespath v0.3.0
go: downloading contrib.go.opencensus.io/exporter/zipkin v0.1.1
go: downloading github.com/Azure/go-autorest v13.4.0+incompatible
go: downloading github.com/Azure/go-autorest/autorest v0.10.2
go: downloading github.com/go-openapi/spec v0.19.6
go: downloading contrib.go.opencensus.io/exporter/ocagent v0.7.1-0.20200907061046-05415f1de66d
go: downloading go.uber.org/atomic v1.6.0
go: downloading golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de
go: downloading github.com/Azure/azure-sdk-for-go v38.2.0+incompatible
go: downloading github.com/dgryski/go-gk v0.0.0-20200319235926-a69029f61654
go: downloading k8s.io/legacy-cloud-providers v0.17.4
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.14.8
go: downloading github.com/streadway/quantile v0.0.0-20150917103942-b0c588724d25
go: downloading github.com/markbates/inflect v1.0.4
go: downloading github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b
go: downloading github.com/Azure/go-autorest/autorest/to v0.3.0
go: downloading golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6
go: downloading github.com/go-openapi/swag v0.19.7
go: downloading github.com/google/go-github/v27 v27.0.6
go: downloading github.com/miekg/dns v1.1.29
go: downloading github.com/docker/cli v0.0.0-20200210162036-a4bedce16568
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/c2h5oh/datasize v0.0.0-20200112174442-28bbd4740fee
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.3.0
go: downloading gonum.org/v1/gonum v0.0.0-20181121035319-3f7ecaa7e8ca
go: downloading gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: downloading github.com/Azure/go-autorest/autorest/adal v0.8.3
go: downloading github.com/tsenart/go-tsz v0.0.0-20180814235614-0bd30b3df1c3
go: downloading github.com/onsi/ginkgo v1.12.1
go: downloading github.com/gobuffalo/envy v1.7.1
go: downloading github.com/onsi/gomega v1.10.1
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/rogpeppe/go-internal v1.5.2
go: downloading k8s.io/klog/v2 v2.0.0
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/Azure/go-autorest/autorest/date v0.2.0
go: downloading github.com/nxadm/tail v1.4.4
go: downloading github.com/dgryski/go-lttb v0.0.0-20180810165845-318fcdf10a77
go: downloading github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: downloading github.com/fsnotify/fsnotify v1.4.9
go: downloading go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/docker/docker v1.13.1
go: downloading gonum.org/v1/netlib v0.0.0-20181029234149-ec6d1f5cefe6
go: downloading github.com/Azure/go-autorest/autorest/validation v0.2.0
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/networking/nscert/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/dataplane-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/deployment-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/load-test/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/rollout-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/scale-from-zero/continuous/kodata/refs
--- Removing unwanted vendor files
--- Updating licenses
go: downloading github.com/google/go-licenses v0.0.0-20200602185517-f29a4c695c3d
go: github.com/google/go-licenses upgrade => v0.0.0-20200602185517-f29a4c695c3d
go: downloading github.com/otiai10/copy v1.2.0
go: downloading golang.org/x/tools v0.0.0-20191118222007-07fc4c7f2b98
go: downloading github.com/google/licenseclassifier v0.0.0-20190926221455-842c0d70d702
go: downloading golang.org/x/sys v0.0.0-20191119060738-e882bf8e40c2
go: downloading golang.org/x/net v0.0.0-20191119073136-fc4aabc6c914
go: downloading golang.org/x/crypto v0.0.0-20191117063200-497ca9f6d64f
/Users/jsoref/code/knative/knative-serving/hack/../vendor/knative.dev/test-infra/scripts/library.sh: line 503: go-licenses: command not found
--- FAIL: go-licenses failed to update licenses

Steps to Reproduce the Problem

Run hack/update-codegen.sh on macOS 10.14.6 (using the branch from #9778 / commit: af905a7ecc78a7bd7a3340151953952f8022919e)

jsoref commented 4 years ago

For reference, here's the change (partial revert of 250b0f220d) that lets this run to the second failure point:

index b11cd33be..ce3b43156 100644
--- a/vendor/knative.dev/test-infra/scripts/library.sh
+++ b/vendor/knative.dev/test-infra/scripts/library.sh
@@ -22,7 +22,7 @@
 readonly KNATIVE_TESTS_PROJECT=knative-tests

 # Conveniently set GOPATH if unset
-if [[ ! -v GOPATH ]]; then
+if [[ -z "${GOPATH:-}" ]]; then
   export GOPATH="$(go env GOPATH)"
   if [[ -z "${GOPATH}" ]]; then
     echo "WARNING: GOPATH not set and go binary unable to provide it"
@@ -30,9 +30,9 @@ if [[ ! -v GOPATH ]]; then
 fi

 # Useful environment variables
-[[ -v PROW_JOB_ID ]] && IS_PROW=1 || IS_PROW=0
+[[ -n "${PROW_JOB_ID:-}" ]] && IS_PROW=1 || IS_PROW=0
 readonly IS_PROW
-[[ ! -v REPO_ROOT_DIR ]] && REPO_ROOT_DIR="$(git rev-parse --show-toplevel)"
+[[ -z "${REPO_ROOT_DIR:-}" ]] && REPO_ROOT_DIR="$(git rev-parse --show-toplevel)"
 readonly REPO_ROOT_DIR
 readonly REPO_NAME="$(basename ${REPO_ROOT_DIR})"

@@ -478,19 +478,24 @@ function start_latest_eventing_sugar_controller() {
 function run_go_tool() {
   local tool=$2
   local install_failed=0
-  if [[ -z "$(which ${tool})" ]]; then
-    local action=get
-    [[ $1 =~ ^[\./].* ]] && action=install
+  local action="get"
+  [[ $1 =~ ^[\./].* ]] && action="install"
+  # Install the tool in the following situations:
+  #   - The tool does not exist.
+  #   - The tool needs to be installed from a local path.
+  #   - Version of the tool is specificied in the given tool path.
+  # TODO(chizhg): derive a better versioning story for the tools being used.
+  if [[ -z "$(which "${tool}")" || "${action}" == "install" || "${tool}" =~ "@" ]]; then
     # Avoid running `go get` from root dir of the repository, as it can change go.sum and go.mod files.
     # See discussions in https://github.com/golang/go/issues/27643.
     if [[ ${action} == "get" && $(pwd) == "${REPO_ROOT_DIR}" ]]; then
       local temp_dir="$(mktemp -d)"
       # Swallow the output as we are returning the stdout in the end.
       pushd "${temp_dir}" > /dev/null 2>&1
-      GOFLAGS="" go ${action} "$1" || install_failed=1
+      GOFLAGS="" go ${action} $1 || install_failed=1
       popd > /dev/null 2>&1
     else
-      GOFLAGS="" go ${action} "$1" || install_failed=1
+      GOFLAGS="" go ${action} $1 || install_failed=1
     fi
   fi
   (( install_failed )) && return ${install_failed}
@@ -587,6 +592,17 @@ function run_kntest() {
   fi
 }

+# Run "kntest" tool
+# Parameters: $1..$n - parameters passed to the tool
+function run_kntest() {
+  if [[ "${REPO_NAME}" == "test-infra" ]]; then
+    go run "${REPO_ROOT_DIR}"/kntest/cmd/kntest "$@"
+  else
+    # Always run the latest version.
+    run_go_tool knative.dev/test-infra/kntest/cmd/kntest@master "$@"
+  fi
+}
+
 # Run go-licenses to update licenses.
 # Parameters: $1 - output file, relative to repo root dir.
 #             $2 - directory to inspect.
@@ -671,7 +687,7 @@ function get_canonical_path() {
 # This is implemented as a function so it can be mocked in unit tests.
 # It will fail if a file name ever contained a newline character (which is bad practice anyway)
 function list_changed_files() {
-  if [[ -v PULL_BASE_SHA ]] && [[ -v PULL_PULL_SHA ]]; then
+  if [[ ! -z "${PULL_BASE_SHA:-}" ]] && [[ ! -z "${PULL_PULL_SHA:-}" ]]; then
     # Avoid warning when there are more than 1085 files renamed:
     # https://stackoverflow.com/questions/7830728/warning-on-diff-renamelimit-variable-when-doing-git-push
     git config diff.renames 0
@@ -724,29 +740,6 @@ function get_latest_knative_yaml_source() {
   echo "https://storage.googleapis.com/knative-nightly/${repo_name}/latest/${yaml_name}.yaml"
 }

-function shellcheck_new_files() {
-  declare -a array_of_files
-  local failed=0
-  readarray -t -d '\n' array_of_files < <(list_changed_files)
-  for filename in "${array_of_files[@]}"; do
-    if echo "${filename}" | grep -q "^vendor/"; then
-      continue
-    fi
-    if file "${filename}" | grep -q "shell script"; then
-      # SC1090 is "Can't follow non-constant source"; we will scan files individually
-      if shellcheck -e SC1090 "${filename}"; then
-        echo "--- PASS: shellcheck on ${filename}"
-      else
-        echo "--- FAIL: shellcheck on ${filename}"
-        failed=1
-      fi
-    fi
-  done
-  if [[ ${failed} -eq 1 ]]; then
-    fail_script "shellcheck failures"
-  fi
-}
-
 # Initializations that depend on previous functions.
 # These MUST come last.
n3wscott commented 4 years ago

Using bash 4 should solve this for you. It is why we use

!/usr/bin/env bash

Vs

!/bin/bash

Several other options are bash 4 only that are in use.

julz commented 4 years ago

looks like we don't currently document this in DEVELOPMENT.md, Ill PR something

jsoref commented 4 years ago

Fwiw, the reason git describe --dirty failed is because out of habit, I remove all branches+tags from my forks before cloning. (I clone a lot and generally don't need branches/tags -- I create enough on my own.)

This would have returned something vaguely useful:

$ git describe --dirty --always
9dc2ab410-dirty
mattmoor commented 3 years ago

I think this has been resolved via @julz documentation update