search

knative / build

A Kubernetes-native Build resource.
Apache License 2.0
575 stars 159 forks source link

v0.4.0 build.yaml file is missing config-logging configmap, controller and webhook deployments #566

Closed khrisrichardson closed 5 years ago

khrisrichardson commented 5 years ago

Expected Behavior

The v0.4.0 build.yaml should presumably include the config-logging configmap and the controller and webhook deployments. Not sure if it's by design, but relative to the v0.3.0 release, definitions of kind image are also missing.

Actual Behavior

Steps to Reproduce the Problem

1. 2. 3.

Additional Info

imjasonh commented 5 years ago

This is a baffling bug.

Running the release script locally produces a release YAML which includes all the Deployments, etc., you'd expect.

The autorelease process that built the 0.4.0 release when it detected the release-0.4 branch did so after passing integration tests, which indicates it's capable of ko resolve-ing config/ into a full and valid config YAML, but it seems like it re-generates the release YAML fresh after passing those tests.

@nbarthwal are there any steps to debug the autorelease process, or even get its logs to see if it said anything useful while it built the release?

nbarthwal commented 5 years ago

Yes, the logs for auto-release can be seen. Since it is a prow job, like any other jobs - you can obtain the logs for the jobs. Locat the jobs you want to look at from: https://prow.knative.dev/?job=ci-knative-build-auto-release https://prow.knative.dev/?job=ci-knative-build-auto-release

And then clicking on it will give the full logs.

I think the logs for the job that build the 0.4.0 should be at : https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-build-auto-release/1098252910535380992/?log#log https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-build-auto-release/1098252910535380992/?log#log

On Feb 22, 2019, at 1:56 PM, Jason Hall notifications@github.com wrote:

This is a baffling bug.

Running the release script locally produces a release YAML which includes all the Deployments, etc., you'd expect.

The autorelease process that built the 0.4.0 release when it detected the release-0.4 branch did so after passing integration tests, which indicates it's capable of ko resolve-ing config/ into a full and valid config YAML, but it seems like it re-generates the release YAML fresh after passing those tests.

@nbarthwal https://github.com/nbarthwal are there any steps to debug the autorelease process, or even get its logs to see if it said anything useful while it built the release?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/knative/build/issues/566#issuecomment-466561767, or mute the thread https://github.com/notifications/unsubscribe-auth/AsL15r4ginZckQ1swrx-emxCaFeZTfcdks5vQGd3gaJpZM4bGr8p.

imjasonh commented 5 years ago

@nbarthwal thanks I'll look through and see if anything stands out.

One thing I think we should fix is that integration tests are run against a config YAML that are then thrown away and regenerated from scratch. AFAICT no integration tests are run against the actual YAML we produce with the release.

imjasonh commented 5 years ago

One curious log line, not sure it's related:

I0220 16:35:17.750] 2019/02/20 16:35:17 error processing import paths in "config/999-cache.yaml": UNKNOWN: Project 'projects/knative-release' not found or deleted.
jonjohnsonjr commented 5 years ago

I'm pretty sure that's supposed to be knative-releases

imjasonh commented 5 years ago

The release didn't build an image for controller (possibly others) for v0.4.0: https://console.cloud.google.com/gcr/images/knative-releases/GLOBAL/github.com/knative/build/cmd/controller

I only see on line indicating an image was published at the end of the release script:

I0220 16:35:17.750] 2019/02/20 16:35:17 Published gcr.io/knative-releases/github.com/knative/build/cmd/nop@sha256:f767af095ef33cca414905fbf0088e1df2df1c92f85de29ba2c91d964a228a3a
imjasonh commented 5 years ago

I'm pretty sure that's supposed to be knative-releases

Yeah. It's a typo, I'm wondering how ko proceeded through that error at all, and I'm wondering if that's why the yaml is incomplete.

imjasonh commented 5 years ago

Re https://github.com/knative/build/issues/566#issuecomment-466574693 the nop image seems to be the only one that was published during the release, but it wasn't even tagged v0.4.0 nor included in the final release yaml.

Current theory is that the knative-release typo caused ko to generate partial yaml and only push some images, but somehow succeed enough to convince the release to proceed.

imjasonh commented 5 years ago

So apparently ko resolve will output whatever yamls it can that don't contain references, then build and push images from the references it finds, then output yaml with those references replaced. The typo in .ko.yaml.release caused ko to barf on that second step, after it had output the non-reference-containing yaml.

Normally this would just cause the release to break because ko resolve failed. Somehow, it didn't, and this line failed but the release proceeded and "succeeded".

I'm not sure how that happened since according to this block it looks like we should be in set -o errexit mode.

imjasonh commented 5 years ago

Fixing the typo and rebasing release-0.4 and re-auto-releasing from that branch created a new v0.4. release which includes all the expected resources.