search

kubernetes / test-infra

Test infrastructure for the Kubernetes project.
Apache License 2.0
3.83k stars 2.64k forks source link

failing to fetch a project from boskos #7925

Closed nikhiljindal closed 6 years ago

nikhiljindal commented 6 years ago

https://k8s-testgrid.appspot.com/sig-multicluster-kubemci#kubemci-ingress-conformance is failing right now. From https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubemci-ingress-conformance/1683/build-log.txt (example log):

W0502 21:55:54.434] 2018/05/02 21:55:54 main.go:749: --gcp-project is missing, trying to fetch a project from boskos.
W0502 21:55:54.434] (for local runs please set --gcp-project to your dev project)
W0502 21:55:54.434] 2018/05/02 21:55:54 main.go:761: provider gce, will acquire project type ingress-project from boskos
W0502 21:55:54.476] 2018/05/02 21:55:54 process.go:93: Saved XML output to /workspace/_artifacts/junit_runner.xml.
W0502 21:55:54.476] 2018/05/02 21:55:54 main.go:305: Something went wrong: failed to prepare test environment: --provider=gce boskos failed to acquire project: resource not found
W0502 21:55:54.477] Traceback (most recent call last):
W0502 21:55:54.478]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 729, in <module>
W0502 21:55:54.478]     main(parse_args())
W0502 21:55:54.478]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 589, in main
W0502 21:55:54.478]     mode.start(runner_args)
W0502 21:55:54.478]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start
W0502 21:55:54.478]     check_env(env, self.command, *args)
W0502 21:55:54.478]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env
W0502 21:55:54.478]     subprocess.check_call(cmd, env=env)
W0502 21:55:54.478]   File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
W0502 21:55:54.478]     raise CalledProcessError(retcode, cmd)
W0502 21:55:54.479] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--provider=gce', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--check-leaked-resources', '--extract=ci/latest', '--gcp-node-image=gci', '--gcp-project-type=ingress-project', '--gcp-zone=us-central1-f', '--test_args=--ginkgo.focus=\\[Feature:kubemci\\] --minStartupPods=8', '--timeout=90m')' returned non-zero exit status 1
E0502 21:55:54.484] Command failed

cc @BenTheElder @krzyzacy @G-Harmon

BenTheElder commented 6 years ago

FWIW this job is failing as far back as testgrid will display with failures not related to boskos.

BenTheElder commented 6 years ago

/assign @krzyzacy /area boskos

nikhiljindal commented 6 years ago

Thanks for pointing that out @BenTheElder

I see the following in logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubemci-ingress-conformance/1355?log#log

event for l7-lb-controller-v1.1.1-bootstrap-e2e-master: {kubelet bootstrap-e2e-master} Failed: Failed to pull image "gcr.io/k8s-ingress-image-push/ingress-gce-e2e-glbc-amd64:latest": rpc error: code = Unknown desc = Error: Status 405 trying to pull repository k8s-ingress-image-push/ingress-gce-e2e-glbc-amd64: "v1 Registry API is disabled. If you are not explicitly using the v1 Registry API, it is possible your v2 image could not be found. Verify that your image is available, or retry with `dockerd --disable-legacy-registry`. See https://cloud.google.com/container-registry/docs/support/deprecation-notices"
I0419 00:45:57.387] Apr 19 00:45:57.385: INFO: At 2018-04-19 00:33:28 +0000 UTC - event for l7-lb-controller-v1.1.1-bootstrap-e2e-master: {kubelet bootstrap-e2e-master} Failed: Error: ImagePullBackOff

Looks like an image pull error. cc @MrHohn did you see the same error in ingress-gce tests? cc @nicksardo as fyi

MrHohn commented 6 years ago

Sorry, seems like we switched from gcr.io/k8s-ingress-image-push/ingress-gce-e2e-glbc-amd64:latest to gcr.io/k8s-ingress-image-push/ingress-gce-e2e-glbc-amd64:master some days ago. The old image may not exist anymore.

Sent #7926.

krzyzacy commented 6 years ago

looks green now, if you run out of ingress-projects, feel free to add more to the pool.

G-Harmon commented 6 years ago

Thanks for the fix. I'm not sure why, but "Test" and "Up" are still red, even though I don't see any associated failed runs...

BenTheElder commented 6 years ago

if you scroll back there are red entries, you may need to configure the num_recent on testgrid.

G-Harmon commented 6 years ago

Okay, where would we set that?

BenTheElder commented 6 years ago

https://github.com/kubernetes/test-infra/tree/master/testgrid#what-counts-as-recent

On Fri, May 4, 2018 at 3:43 PM G-Harmon notifications@github.com wrote:

Okay, where would we set that?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/test-infra/issues/7925#issuecomment-386753244, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4Bq58zzz-Y6hUbenkkIRsGViMcyIdhks5tvNl-gaJpZM4TwTYi .