Testing fail after integration with Kubeflow

[aym-frikha]

Hello,

I'm deploying this bundle kubeflow + seldon + knative. Deployed this knative event And this seldon example

This is the output of kn broker list This is the output of kn service list

this is the k8s resources

I did a port forward on seldon pod: kubectl port-forward seldon2-model-example2-0-classifier-f44b9c748-9ktqr -n default :8000 I sent a prediction request which return correctly an output

But the broker messages is empty. No event shows up

[aym-frikha]

After further troubleshooting I found those logs in the knative eventing namespace:

[mt-broker-filter-8799894b4-6c8bm] {"level":"error","ts":"2022-07-29T15:31:05.537Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:205","msg":"failed to send event","commit":"b3bccaf","error":"failed to dispatch message: Post \"http://cloudevents-player.default.svc.cluster.local\": dial tcp 10.152.183.94:80: i/o timeout","stacktrace":"knative.dev/eventing/pkg/broker/filter.(*Handler).send\n\tknative.dev/eventing/pkg/broker/filter/filter_handler.go:205\nknative.dev/eventing/pkg/broker/filter.(*Handler).ServeHTTP\n\tknative.dev/eventing/pkg/broker/filter/filter_handler.go:198\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\tgo.opencensus.io@v0.23.0/plugin/ochttp/server.go:92\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/network/handlers/drain.go:110\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2878\nnet/http.(*conn).serve\n\tnet/http/server.go:1929"} [imc-dispatcher-b8bf96b6d-v9mlm] {"level":"error","ts":"2022-07-29T15:31:05.537Z","logger":"inmemorychannel-dispatcher","caller":"fanout/fanout_message_handler.go:281","msg":"Fanout had an error","commit":"b3bccaf","knative.dev/pod":"imc-dispatcher-b8bf96b6d-v9mlm","knative.dev/controller":"knative.dev.eventing.pkg.reconciler.inmemorychannel.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.InMemoryChannel","knative.dev/traceid":"3dab1f57-7326-45af-95c2-3938b5760435","knative.dev/key":"default/bpk-broker-kne-trigger","error":"unable to complete request to http://broker-filter.knative-eventing.svc.cluster.local/triggers/default/bpk-broker-trigger/5ce90948-7276-42fe-9dd4-ac90979ee523: unexpected HTTP response, expected 2xx, got 500","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*FanoutMessageHandler).dispatch\n\tknative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:281\nknative.dev/eventing/pkg/channel/fanout.createMessageReceiverFunction.func1.1\n\tknative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:191"}

[DnPlas]

Hi @aym-frikha, please correct me if I am wrong, your SeldonDeployment is configured so that on each prediction, it logs some data that is sent to the broker as a CloudEvent. I have tried to reproduce this issue, and I can see that Seldon is not able to send CloudEvents correctly to the URL that you have specified in the deployment:

$ kubectl logs seldon-model-example-0-classifier-86476b8d5c-slx8z -c seldon-container-engine

Work request queued
worker4: Received work request for http://broker-ingress.knative-eventing.svc.cluster.local/data-drift/bpk-broker
{"level":"error","ts":1659473091.7163737,"logger":"entrypoint","msg":"Failed to send log","URL":"http://broker-ingress.knative-eventing.svc.cluster.local/data-drift/bpk-broker","error":"while sending event: error sending cloudevent: 500 Internal Server Error","stacktrace":"github.com/seldonio/seldon-core/executor/logger.(*Worker).Start.func1\n\t/workspace/logger/worker.go:124"}

This makes me think if there's something missing in the configuration on either seldon or the broker itself.

[aym-frikha]

This is right @DnPlas, but when I changed the url of the broker on seldon deployment configuration to "http://broker-ingress.knative-eventing.svc.cluster.local/default/bpk-broker" (%s/data-drift/default), it was able to send the request to the broker, but I had the issue in the broker log that says, it's not able to access the DNS server on port 53 (CoreDNS).

[lolwww]

Seeing the same issue (empty broker message) with the fix from Daniela.

I'm using

Bundle cloudevents-player seldon

After I curl curl -s /api/v0.1/predictions.. I can get the response like _{"data":{"names":["proba"],"ndarray":[[0.8446838498233856],[0.8446838498233856]]},"meta":{"requestPath":{"classifier":"seldonio/mockclassifier:1.13.1"}}}

However later when I try to curl /messages I get empty list []

Am I missing something?