Eventing-0.5 and eventing-sources-0.5 tests are consistently failing

chizhg commented 5 years ago

These two jobs are consistently failing. https://testgrid.knative.dev/knative-eventing-0.5#continuous https://testgrid.knative.dev/knative-eventing-sources-0.5#continuous

In the logs, we can see the following error:

I0430 08:51:06.371] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "attributemanifest" in version "config.istio.io/v1alpha2"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "attributemanifest" in version "config.istio.io/v1alpha2"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "handler" in version "config.istio.io/v1alpha2"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "rule" in version "config.istio.io/v1alpha2"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "rule" in version "config.istio.io/v1alpha2"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "kubernetes" in version "config.istio.io/v1alpha2"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
I0430 08:51:06.372] unable to recognize "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
I0430 08:51:06.379] ERROR: Knative setup failed

adrcunha commented 5 years ago

Sounds like infra, please have the oncall investigate.

adrcunha commented 5 years ago

@steuhs can you please investigate?

adrcunha commented 5 years ago

/assign @steuhs

steuhs commented 5 years ago

Knative serving stopped publishing istio images starting version 0.6. However, the two jobs mentioned above are trying to use the yaml from the latest (0.6) nightly, "https://storage.googleapis.com/knative-nightly/serving/latest/istio.yaml". Letting the jobs point to 0.5 istio yaml can potentially solve the problem.

/assign @Fredy-Z

srinivashegde86 commented 5 years ago

/uncc @steuhs

chizhg commented 5 years ago

As of now, we did some changes to handle this issue in test-infra, but it still has a problem.

Our current solution is: under the repository's root folder, execute $(git describe --tags --abbrev=0) to get the latest tag name for the current branch, and then use it to retrieve the latest release build url of other repositories. With this logic, we assume that all repositories have the same latest tag, but it's not the case.

For example, the latest tag for eventing-sources release-0.5 is v0.5.1. But for eventing release-0.5, it is v0.5.0. So when we try to get https://storage.googleapis.com/knative-releases/eventing/previous/v0.5.1/release.yaml for eventing with the above logic, we will get a 404 not found error.

srinivashegde86 commented 5 years ago

can we merge the chanage for installing istio through add-on in eventing-0.5 and eventing-sources-0.5? https://github.com/knative/test-infra/pull/734 and https://github.com/knative/test-infra/pull/739

chizhg commented 5 years ago

eventing-0.5 does not work with the istio add-on, it can only work with the istio-0.5 shipped together with serving-0.5, see https://github.com/knative/eventing/pull/1253.

adrcunha commented 5 years ago

Looks like you have to make get_knative_base_yaml_source() module-specific for branches. Basically:

on a master branch, use nightly, problem solved;
on a release branch, (a) get the branch version; (b) for each particular module, identify the latest version of that particular branch by inspecting gs://knative-releases; (c) if you can't pair [a] with [b], use the latest version before [a];

adrcunha commented 5 years ago

The fix needs to be propagated to the affected branches.

adrcunha commented 5 years ago

eventing-0.5 is now passing: https://storage.googleapis.com/knative-prow/logs/ci-knative-eventing-0.5-continuous/1135828755776802816/build-log.txt

eventing-sources-0.5 is now passing: https://prow.knative.dev/log?job=ci-knative-eventing-sources-0.5-continuous&id=1135986043321847808

knative / test-infra

Eventing-0.5 and eventing-sources-0.5 tests are consistently failing #723