search

SeldonIO / seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
https://www.seldon.io/tech/products/core/
Other
4.4k stars 832 forks source link

opentelemetry enable ignored in seldon-core-v2-setup chart #4962

Open franco-bocci opened 1 year ago

franco-bocci commented 1 year ago

The field opentelemetry.enable is being ignored in the seldon-core-v2-setup chart. https://github.com/SeldonIO/helm-charts/blob/master/helm-charts/seldon-core-v2-setup/values.yaml#L74

This causes errors at least in the dataflow-engine component like:

[otel.javaagent 2023-06-23 09:58:28:834 +0000] [OkHttp http://seldon-collector:4317/...] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: seldon-collector: Name or service not known

RafalSkolasinski commented 1 year ago

hi, please raise the issue on seldon core repository https://github.com/SeldonIO/seldon-core/

RafalSkolasinski commented 1 year ago

actually - I was able to transfer it

Kolajik commented 1 year ago

The field was renamed to opentelemetry.disable and it still does not work if this field is set to true. I get errors from multiple resources within Seldon Core v2 with log:

2023/11/07 14:13:30 max retry time elapsed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup seldon-collector.seldon-mesh on 169.254.20.10:53: no such host"

The images and tags I use are:

hpadakanti0618 commented 9 months ago

Can i know what is the fix or solution here ? i am using 2.6.0 version libraries and i get the same issue

[otel.javaagent 2023-06-23 09:58:28:834 +0000] [OkHttp http://seldon-collector:4317/...] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: seldon-collector: Name or service not known

And instead seldon-mesh mamespace. we have our own name space. I should change the endpoint right as per my new namespace name like below ?

        "opentelemetry": map[string]interface{}{
            "endpoint": "seldon-collector.<my-own-ns>:4317",
        },