Github actions e2e tests with istio-ambient are very flaky

knative / serving

Kubernetes-based, scale-to-zero, request-driven compute

https://knative.dev/docs/serving/

Apache License 2.0

5.51k stars 1.15k forks source link

Github actions e2e tests with istio-ambient are very flaky #14637

Open ReToCode opened 9 months ago

ReToCode commented 9 months ago

Examples

Also take a look at the discussion here: https://github.com/knative/serving/pull/14610#issuecomment-1893894692

dprotaso commented 9 months ago

@nak3 @ReToCode did anything change recently with the ambient configuration?

dprotaso commented 9 months ago

Maybe Istio 1.20?

https://github.com/knative/serving/pull/14629

nak3 commented 9 months ago

I'm sure the flake started before 1.20. I bumped 1.20 if the flake can be fixed or not after getting a ping on slack https://cloud-native.slack.com/archives/C04LMU0AX60/p1700061890467089

The change may be affected is bumping to v1.9.3 https://github.com/knative-extensions/net-istio/pull/1194 https://github.com/knative-extensions/net-istio/pull/1196

nak3 commented 9 months ago

Today's github action does not hit the flake at all.

Serving repo - https://github.com/knative/serving/actions/runs/6934043757

I have tried 3 times on my repo but no error happened.

It seems that infra (github action) level issue?

ReToCode commented 9 months ago

Well that is interesting. @nak3 your environment with GH actions should not be different, right? So I have no idea why this works well and then doesn't like in https://github.com/knative/serving/pull/14627.

nak3 commented 9 months ago

Well that is interesting. @nak3 your environment with GH actions should not be different, right?

Yes, no different and actually it failed yesterday.

ReToCode commented 9 months ago

Hm, so still disabling it until we figure it out?

nak3 commented 9 months ago

Yeah, I'm ok with disabling it.