NSM in kubeadm cluster

[p4lik4ri]

Question

Hello i am using the basic examples for the NSM in a kind cluster which they work fine, but when i try the same examples on my "production" cluster, some of the pods are stack in the init state or even when they are running they are not working properly.

Examples

Lets try an example of kernel to IP to kernel connection on a Kind cluster.

ns-kernel2ip2kernel   alpine                                       2/2     Running   0              
ns-kernel2ip2kernel   nse-kernel-5787687848-5dc4z                  1/1     Running   0              

I can ping from the NSC to NSE and vice versa without a problem.

On the other hand, in my "production" cluster, the alpine pod is stack on the init state.

ns-kernel2ip2kernel   alpine                                         0/2     Init:0/1                 
ns-kernel2ip2kernel   nse-kernel-78dd9c4dfd-rq42f                    1/1     Running 

The logs of the alpine pod are the following:

kubectl logs -n ns-kernel2ip2kernel alpine cmd-nsc-init:
This application is configured via the environment. The following environment
variables can be used:

KEY                            TYPE                                                   DEFAULT                                                REQUIRED    DESCRIPTION
NSM_NAME                       String                                                 cmd-nsc-init                                                       Name of the client
NSM_DIAL_TIMEOUT               Duration                                               5s                                                                 timeout to dial NSMgr
NSM_REQUEST_TIMEOUT            Duration                                               15s                                                                timeout to request NSE
NSM_CONNECT_TO                 URL                                                    unix:///var/lib/networkservicemesh/nsm.io.sock                     url to connect to
NSM_MAX_TOKEN_LIFETIME         Duration                                               10m                                                                maximum lifetime of tokens
NSM_NETWORK_SERVICES           Comma-separated list of URL                                                                                               A list of Network Service Requests
NSM_AWARENESS_GROUPS           Comma-separated list of Comma-separated list of URL                                                                       Awareness groups for mutually aware NSEs
NSM_LOG_LEVEL                  String                                                 INFO                                                               Log level
NSM_OPEN_TELEMETRY_ENDPOINT    String                                                 otel-collector.observability.svc.cluster.local:4317                OpenTelemetry Collector Endpoint
NSM_METRICS_EXPORT_INTERVAL    Duration                                               10s                                                                interval between mertics exports
time="2024-05-16T22:28:36Z" level=info msg="Starting NetworkServiceMesh Client ..."
May 16 22:28:36.935 [INFO] [cmd:[/bin/app]] rootConf: &{Name:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0 DialTimeout:5s RequestTimeout:15s ConnectTo:{Scheme:unix Opaque: User: Host: Path:/var/lib/networkservicemesh/nsm.io.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:} MaxTokenLifetime:10m0s NetworkServices:[{Scheme:kernel Opaque: User: Host:kernel2ip2kernel Path:/nsm-1 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}] AwarenessGroups:[] LogLevel:TRACE OpenTelemetryEndpoint:otel-collector.observability.svc.cluster.local:4317 MetricsExportInterval:10s}
May 16 22:28:38.177 [INFO] [cmd:[/bin/app]] sVID: "spiffe://k8s.nsm/ns/ns-kernel2ip2kernel/pod/alpine"
May 16 22:28:38.177 [INFO] [cmd:[/bin/app]] NSC: Connecting to Network Service Manager unix:///var/lib/networkservicemesh/nsm.io.sock
May 16 22:28:38.188 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:28:38.188 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1)   request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2)   request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:28:38.189 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1)      Environment variable NODE_NAME is not set. Skipping.
May 16 22:28:38.189 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2)      Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3)      request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5)     ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1)         request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1)          request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9)         ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1)            request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12)            ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12.1)              rpc error: code = DeadlineExceeded desc = context deadline exceeded;        Error returned from api/pkg/api/networkservice/networkServiceClient.Request;    github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.logError;              /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/common.go:211;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:57;github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;  github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/excludedprefixes/client.go:115;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;          /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/mechanisms/sendfd/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request;          /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/authorize/client.go:77;     github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;
May 16 22:28:53.188 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.1)             request-response=null
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.2)             Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.2)            Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9.1)           Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.2)          Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.2)         Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6.1)        Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5.1)       Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.4)      Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [DEBU] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [connID:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [metadata:client] [type:networkService] (3.1)     metadata deleted
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3.2)     Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2.1)    Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.3)   Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1)   request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2)   request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:28:53.396 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1)      Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:28:53.396 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2)      Environment variable NODE_NAME is not set. Skipping.
May 16 22:28:53.397 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3)      request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:28:53.403 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5)     ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:28:53.403 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:28:53.403 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:28:53.405 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1)         request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:28:53.405 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:28:53.406 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1)          request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:28:53.406 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9)         ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:28:53.406 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:28:53.412 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1)            request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:28:53.412 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:28:53.412 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12)            ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12.1)              rpc error: code = DeadlineExceeded desc = context deadline exceeded;        Error returned from api/pkg/api/networkservice/networkServiceClient.Request;    github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.logError;              /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/common.go:211;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:57;github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;  github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/excludedprefixes/client.go:115;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;          /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/mechanisms/sendfd/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request;          /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/authorize/client.go:77;     github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;
May 16 22:29:08.393 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.1)             request-response=null
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.2)             Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.2)            Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9.1)           Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.2)          Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.2)         Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6.1)        Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5.1)       Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.4)      Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [DEBU] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [connID:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [metadata:client] [type:networkService] (3.1)     metadata deleted
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3.2)     Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2.1)    Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.3)   Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.412 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1)   request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2)   request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:29:08.613 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1)      Environment variable NODE_NAME is not set. Skipping.
May 16 22:29:08.613 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2)      Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3)      request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5)     ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1)         request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1)          request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9)         ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1)            request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12)            ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12.1)              rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed; Error returned from api/pkg/api/networkservice/networkServiceClient.Request;github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.logError;          /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/common.go:211;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:57; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/excludedprefixes/client.go:115;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;             /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/mechanisms/sendfd/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;  /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/common/authorize/client.go:77;    github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/client.go:49;      github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20240418094831-608b1b518564/pkg/networkservice/core/next/client.go:60;
May 16 22:29:23.617 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.1)             request-response=null
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.2)             Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.2)            Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9.1)           Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.2)          Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.2)         Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6.1)        Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5.1)       Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.4)      Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [DEBU] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [connID:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [metadata:client] [type:networkService] (3.1)     metadata deleted
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3.2)     Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2.1)    Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.3)   Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1)   request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2)   request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:29:23.819 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1)      Environment variable NODE_NAME is not set. Skipping.
May 16 22:29:23.819 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2)      Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3)      request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5)     ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1)         request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1)          request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9)         ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1)            request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12)            ⎆ api/pkg/api/networkservice/networkServiceClient.Request()

Context

the same error logs are similar with other examples, i suspect that it has to do that i use eBPF instead of kube-proxy (iptables)?

• Kubernetes Version: kubeadm 1.28.3 - ubuntu 22.0.4
• 1 master, 2 worker nodes
• CNI: Cilium without kube-proxy