search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.58k stars 788 forks source link

step-build-container-build container of stage everything-else fails for Amazon ECR #6441

Closed slayerjain closed 4 years ago

slayerjain commented 4 years ago

Summary

I've setup JX on a EKS cluster (and I'm using Amazon ECR). I tried creating a quickstart app (jx create quickstart ) - golang-http-from-jenkins-x-yml. The creation setup seems fine (it also creates git repos, adds to dep matrix of the environments, and also creates docker repo on Amazon ECR), but doesn't create the app. step-build-container-build seems to pushing to GCE instead of the configured Amazon ECR.

Steps to reproduce the behavior

Setup JX on Amazon EKS and use Amazon ECR as the docker registry. Then try creating a quickstart app.

Expected behavior

Quickstart app build and deployed.

Actual behavior

jx get activity -f jx-demo-go-2 -w
STEP                                                  STARTED AGO DURATION STATUS
slayerjain/jx-demo-go-2/master #1                            4m0s     1m4s Failed
  meta pipeline                                              4m0s      19s Succeeded
    Credential Initializer C9cvd                             4m0s       0s Succeeded
    Working Dir Initializer Zzsnn                            4m0s       0s Succeeded
    Place Tools                                              4m0s       1s Succeeded
    Git Source Meta Slayerjain Jx Demo Go 2 Ma Zkxqt        3m59s       5s Succeeded https://github.com/slayerjain/jx-demo-go-2.git
    Git Merge                                               3m54s       1s Succeeded
    Merge Pull Refs                                         3m53s       0s Succeeded
    Create Effective Pipeline                               3m53s       4s Succeeded
    Create Tekton Crds                                      3m49s       8s Succeeded
  run make                                                  3m24s      19s Succeeded
    Credential Initializer 9rmhl                            3m24s       0s Succeeded
    Working Dir Initializer Rb9gs                           3m24s       1s Succeeded
    Place Tools                                             3m23s       1s Succeeded
    Create Dir Workspace Sktg7                              3m22s       5s Succeeded
    Git Source Slayerjain Jx Demo Go 2 Master Bpqlt         3m17s       1s Succeeded https://github.com/slayerjain/jx-demo-go-2.git
    Git Merge                                               3m16s       1s Succeeded
    Setup Jx Git Credentials                                3m15s       0s Succeeded
    Build Make Build                                        3m15s       9s Succeeded
    Source Mkdir Slayerjain Jx Demo Go 2 Master Fp2sj        3m6s       0s Succeeded
    Source Copy Slayerjain Jx Demo Go 2 Master Cq4g2         3m6s       1s Succeeded
  everything else                                            3m3s       7s Failed
    Credential Initializer Xckp6                             3m3s       0s Succeeded
    Working Dir Initializer C8bs6                            3m3s       0s Succeeded
    Place Tools                                              3m3s       1s Succeeded
    Create Dir Workspace Twjp8                               3m2s       4s Succeeded
    Source Copy Workspace 26zs2                             2m58s       1s Succeeded
    Build Container Build                                   2m57s       0s Failed
    Build Post Build                                        2m57s       1s NotExecuted
    Promote Changelog                                       2m56s       0s NotExecuted
    Promote Helm Release                                    2m56s       0s NotExecuted
    Promote Jx Promote                                      2m56s       0s NotExecuted
jx get build logs slayerjain/jx-demo-go-2/master
wrote: /var/folders/nv/rcbllzwj6cg01rqf675b57p00000gn/T/50014e19-5932-42c1-a669-e31e3657b6a8276147074

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-credential-initializer-9rmhl
{"level":"warn","ts":1578466192.7143176,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-**-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1578466192.7146554,"logger":"fallback-logger","caller":"creds-init/main.go:40","msg":"Credentials initialized."}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-working-dir-initializer-rb9gs
{"level":"warn","ts":1578466193.7465518,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466193.7474716,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/source\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-place-tools

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-create-dir-workspace-sktg7
{"level":"warn","ts":1578466199.8445256,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466199.8452456,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/source\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-git-source-slayerjain-jx-demo-go-2-master-bpqlt
{"level":"warn","ts":1578466200.0560741,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-**-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1578466200.6163392,"logger":"fallback-logger","caller":"git/git.go:103","msg":"Successfully cloned https://github.com/**********/**-demo-go-2.git @ v0.0.1 in path /workspace/source"}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-git-merge
Using SHAs from PULL_REFS=master:2bd773bbb84f42a4bd90ed849536ea099ebe0e69
WARNING: no SHAs to merge, falling back to initial cloned commit

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-setup-jx-git-credentials
Generated Git credentials file /workspace/xdg_config/git/credentials

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-build-make-build
CGO_ENABLED=0 GO15VENDOREXPERIMENT=1 go build -ldflags '' -o bin/**-demo-go-2 main.go

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-source-mkdir-slayerjain-jx-demo-go-2-master-fp2sj
{"level":"warn","ts":1578466210.865869,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466210.867573,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /pvc/run-make/workspace\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage run-make and container step-source-copy-slayerjain-jx-demo-go-2-master-cq4g2
{"level":"warn","ts":1578466211.0683136,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466211.0790234,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c cp -r /workspace/source/. /pvc/run-make/workspace\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage everything-else and container step-credential-initializer-xckp6
{"level":"warn","ts":1578466213.3709867,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-**-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1578466213.3714645,"logger":"fallback-logger","caller":"creds-init/main.go:40","msg":"Credentials initialized."}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage everything-else and container step-working-dir-initializer-c8bs6
{"level":"warn","ts":1578466213.7108212,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466213.711489,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/source\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage everything-else and container step-place-tools

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage everything-else and container step-create-dir-workspace-twjp8
{"level":"warn","ts":1578466218.770997,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466218.7716398,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/source\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage everything-else and container step-source-copy-workspace-26zs2
{"level":"warn","ts":1578466218.967071,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1578466218.9752817,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c cp -r /pvc/run-make/workspace/. /workspace/source\"; output "}

Showing logs for build slayerjain-jx-demo-go-2-master-69gjl-1 stage everything-else and container step-build-container-build
INFO[0000] No base image, nothing to extract
INFO[0000] cmd: EXPOSE
INFO[0000] Adding exposed port: 8080/tcp
INFO[0000] Using files from context: [/workspace/source/bin]
INFO[0000] Skipping unpacking as no commands require it.
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] EXPOSE 8080
INFO[0000] cmd: EXPOSE
INFO[0000] Adding exposed port: 8080/tcp
INFO[0000] No files changed in this command, skipping snapshotting.
INFO[0000] ENTRYPOINT ["/**-demo-go-2"]
INFO[0000] No files changed in this command, skipping snapshotting.
INFO[0000] Using files from context: [/workspace/source/bin]
INFO[0000] COPY ./bin/ /
INFO[0000] Taking snapshot of files...
error pushing image: failed to push to destination gcr.io/**********/**-demo-go-2:0.0.1: Get https://gcr.io/v2/token?scope=repository%3A**********%2F**-demo-go-2%3Apush%2Cpull&service=gcr.io: invoking docker-credential-gcr: exit status 1; output: docker-credential-gcr/helper: could not retrieve GCR's access token: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /kaniko-secret/secret.json: no such file or directory

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.1103
Kubernetes cluster v1.14.9-eks-c0eccc
kubectl            v1.16.3
helm client        Client: v2.16.1+gbbdfe5e
git                2.24.0
Operating System   Mac OS X 10.15.2 build 19C57

verifying packages

Jenkins type

Kubernetes cluster

Amazon EKS cluster.

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:29Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Operating system / Environment

macOS Catalina

jstrachan commented 4 years ago

what does your jx-requirement.yml look like? you have set the provider: eks right?

slayerjain commented 4 years ago

Yes. Without that set jx boot logged errors and didn't proceed.

Here's the jx-requirements.yml

autoUpdate:
  enabled: false
  schedule: ""
bootConfigURL: https://github.com/slayerjain/environment-staging-cluster-dev
cluster:
  clusterName: staging-cluster
  environmentGitOwner: slayerjain
  gitKind: github
  gitName: github
  gitServer: https://github.com
  namespace: jx
  provider: eks
  region: us-west-2
  registry: https://some.dkr.ecr.us-west-2.amazonaws.com
environments:
- ingress:
    domain: <ip>.nip.io
    externalDNS: false
    namespaceSubDomain: -jx.
    tls:
      email: ""
      enabled: false
      production: false
  key: dev
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: staging
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: production
gitops: true
ingress:
  domain: <ip>.nip.io
  externalDNS: false
  namespaceSubDomain: -jx.
  tls:
    email: ""
    enabled: false
    production: false
kaniko: true
repository: nexus
secretStorage: local
storage:
  backup:
    enabled: false
    url: ""
  logs:
    enabled: false
    url: ""
  reports:
    enabled: false
    url: ""
  repository:
    enabled: false
    url: ""
vault: {}
velero: {}
versionStream:
  ref: v1.0.285
  url: https://github.com/jenkins-x/jenkins-x-versions.git
webhook: prow
slayerjain commented 4 years ago

@jstrachan If you can guide me, I'd be delighted to solve this bug :)

hferentschik commented 4 years ago

@dgozalo, wdyt ^^?

slayerjain commented 4 years ago

@hferentschik So the issue came from because of the GCR hardcoding in https://github.com/jenkins-x-quickstarts/golang-http-from-jenkins-x-yml. I tried another quickstart which didn't have the jenkins-x.yml file explicitly define the repo, there I had problems with authentication.

dgozalo commented 4 years ago

Exactly, some of our quickstarts contain a jenkins-x file that has a hardcoded reference to GCR, so we'd have to make them generic to work with ECR.

@slayerjain Did you create the cluster with eksctl? Right now we are only creating permissions for clusters created by eksctl as it's the officially supported tool for EKS.

slayerjain commented 4 years ago

Yes, I provisioned the cluster via eksctl myself.

deanesmith commented 4 years ago

@slayerjain This recent document may help you better configure your cloud resources and permissions - https://jenkins-x.io/docs/getting-started/setup/boot/clouds/amazon/

jenkins-x-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

jenkins-x-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten

jenkins-x-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten. Provide feedback via https://jenkins-x.io/community. /close

jenkins-x-bot commented 4 years ago

@jenkins-x-bot: Closing this issue.

In response to [this](https://github.com/jenkins-x/jx/issues/6441#issuecomment-663152341): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. >Provide feedback via https://jenkins-x.io/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository.