search

kyma-project / kyma

Kyma is an opinionated set of Kubernetes-based modular building blocks, including all necessary capabilities to develop and run enterprise-grade cloud-native applications.
https://kyma-project.io
Apache License 2.0
1.52k stars 405 forks source link

Application-operator / application-connector tests fail with HTTP 500 from event-service #8133

Closed sayanh closed 4 years ago

sayanh commented 4 years ago

Description

Application-operator/application-connector test fails sometimes with event-service giving with HTTP 500. Error:

2020-04-21 09:58:08.171 CEST
{"ts":"2020-04-21T07:58:08.171Z","sink":"http://operator-app-test-44pt-kn-channel.kyma-integration.svc.cluster.local","stacktrace":"github.com/kyma-project/kyma/components/event-sources/adapter/http.(*httpAdapter).serveHTTP\n\t/workspace/go/src/github.com/kyma-project/kyma/components/event-sources/a…
Expand all | Collapse all{
 insertId: "rub22hoqmv27jhncr"  
 jsonPayload: {
  caller: "http/adapter.go:167"   
  error: "error sending cloudevent: 500 Internal Server Error"   
  level: "error"   
  logger: "http-source"   
  msg: "failed to send cloudevent to sink"   
  sink: "http://operator-app-test-44pt-kn-channel.kyma-integration.svc.cluster.local"   
  stacktrace: "github.com/kyma-project/kyma/components/event-sources/adapter/http.(*httpAdapter).serveHTTP
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/adapter/http/adapter.go:167
reflect.Value.call
    /usr/local/go/src/reflect/value.go:447
reflect.Value.Call
    /usr/local/go/src/reflect/value.go:308
github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/client.(*receiverFn).invoke
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/client/receiver.go:93
github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/client.(*ceClient).obsReceive
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/client/client.go:132
github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/client.(*ceClient).Receive
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/client/client.go:120
github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http.(*Transport).obsInvokeReceiver
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http/transport.go:500
github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http.(*Transport).invokeReceiver
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http/transport.go:484
github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http.(*Transport).ServeHTTP
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/vendor/github.com/cloudevents/sdk-go/pkg/cloudevents/transport/http/transport.go:592
net/http.(*ServeMux).ServeHTTP
    /usr/local/go/src/net/http/server.go:2375
github.com/kyma-project/kyma/components/event-sources/adapter/http.readinessMiddleware.ServeHTTP
    /workspace/go/src/github.com/kyma-project/kyma/components/event-sources/adapter/http/adapter.go:129
net/http.serverHandler.ServeHTTP
    /usr/local/go/src/net/http/server.go:2774
net/http.(*conn).serve
    /usr/local/go/src/net/http/server.go:1878"   
  ts: "2020-04-21T07:58:08.171Z"   
 }
 labels: {…}  
 logName: "projects/sap-kyma-prow-workloads/logs/stdout"  
 receiveTimestamp: "2020-04-21T07:58:21.213133001Z"  
 resource: {…}  
 severity: "INFO"  
 timestamp: "2020-04-21T07:58:08.171827436Z"  
}

Logs: https://console.cloud.google.com/logs/viewer?authuser=1&project=sap-kyma-prow-workloads&minLogLevel=0&expandAll=false&timestamp=2020-04-21T08%3A03%3A55.346281638Z&customFacets&limitCustomFacetWidth=true&interval=JUMP_TO_TIME&scrollTimestamp=2020-04-21T08%3A03%3A55.346281638Z&angularJsUrl=%2Flogs%2Fviewer%3Fauthuser%3D1%26hl%3Den%26project%3Dsap-kyma-prow-workloads%26minLogLevel%3D0%26expandAll%3Dfalse%26timestamp%3D2020-04-20T06%253A03%253A57.000000000Z%26customFacets%26limitCustomFacetWidth%3Dtrue%26interval%3DJUMP_TO_TIME%26scrollTimestamp%3D2020-04-20T11%253A22%253A21.980524365Z%26advancedFilter%3Dresource.type%253D%2522k8s_container%2522%250Aresource.labels.container_name%253D%2522tests%2522%250Aresource.labels.namespace_name%253D%2522kyma-integration%2522%250Aresource.labels.location%253D%2522europe-west4%2522%250Aresource.labels.project_id%253D%2522sap-kyma-prow-workloads%2522%250Aresource.labels.pod_name%253D%2522oct-tp-testsuite-all-2020-04-20-11-12-application-operator-0%2522%250Aresource.labels.cluster_name%253D%2522weekly%2522%250Atimestamp%253D%25222020-04-20T11%253A22%253A21.980514941Z%2522%250AinsertId%253D%2522rub22hoqmv27hdbww%2522%26dateRangeUnbound%3DforwardInTime&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.container_name%3D%22user-container%22%0Aresource.labels.namespace_name%3D%22kyma-integration%22%0Aresource.labels.location%3D%22europe-west4%22%0Aresource.labels.project_id%3D%22sap-kyma-prow-workloads%22%0Aresource.labels.cluster_name%3D%22weekly%22%0Aresource.labels.pod_name%3D%22operator-app-test-44pt-hqpk6-deployment-5bd7b58999-fmztc%22%0Atimestamp%3D%222020-04-21T07%3A58%3A08.171827436Z%22%0AinsertId%3D%22rub22hoqmv27jhncr%22&dateRangeUnbound=forwardInTime

Expected result

event-service should return HTTP 200.

Actual result

event-service returns HTTP 500.

Steps to reproduce

Happens rarely. Not sure how to reproduce.

Troubleshooting

Kyma: 1.11.0+ (eu.gcr.io/kyma-project/kyma-installer:master-011e62d0) Stability checker in weekly cluster.

sayanh commented 4 years ago

@Szymongib
I think the pattern of the application-operator and application-connector tests are not robust enough. Here’s what we do:

We already have a solution in test-core-external-solution-integration https://github.com/kyma-project/kyma/blob/master/tests/end-to-end/external-solution-integration/pkg/testsuite/create_application.go#L84

I think both these tests should be adapted to include the readiness check for the event-source(httpsource)

stale[bot] commented 4 years ago

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

janmedrek commented 4 years ago

Not popped up since April, closing this issue.