Kernel2IP2Kernel alpine pod cannot start in production environment

[316953425]

hi @glazychev-art and other developers

I really like the nsm project, it feels like an exciting project and I'm in production this week， I deploy deployments-k8s version 1.11.0, and then deploy the officially provided Kernel2IP2Kernel example, but the alpine pod cannot be started. The error message is as follows:

[root@CNCP-MS-01 use-cases]# kubectl logs -f alpine -n ns-kernel2ip2kernel -c cmd-nsc-init
time="2023-11-24T06:58:54Z" level=info msg="Starting NetworkServiceMesh Client ..."
This application is configured via the environment. The following environment
variables can be used:

KEY                            TYPE                                                   DEFAULT                                                REQUIRED    DESCRIPTION
NSM_NAME                       String                                                 cmd-nsc-init                                                       Name of the client
NSM_DIAL_TIMEOUT               Duration                                               5s                                                                 timeout to dial NSMgr
NSM_REQUEST_TIMEOUT            Duration                                               15s                                                                timeout to request NSE
NSM_CONNECT_TO                 URL                                                    unix:///var/lib/networkservicemesh/nsm.io.sock                     url to connect to
NSM_MAX_TOKEN_LIFETIME         Duration                                               10m                                                                maximum lifetime of tokens
NSM_NETWORK_SERVICES           Comma-separated list of URL                                                                                               A list of Network Service Requests
NSM_AWARENESS_GROUPS           Comma-separated list of Comma-separated list of URL                                                                       Awareness groups for mutually aware NSEs
NSM_LOG_LEVEL                  String                                                 INFO                                                               Log level
NSM_OPENTELEMETRYENDPOINT      String                                                 otel-collector.observability.svc.cluster.local:4317                OpenTelemetry Collector Endpoint
NSM_METRICS_EXPORT_INTERVAL    Duration                                               10s                                                                interval between mertics exports
Nov 24 06:58:54.600 [INFO] [cmd:[/bin/app]] rootConf: &{Name:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4 DialTimeout:5s RequestTimeout:15s ConnectTo:{Scheme:unix Opaque: User: Host: Path:/var/lib/networkservicemesh/nsm.io.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:} MaxTokenLifetime:10m0s NetworkServices:[{Scheme:kernel Opaque: User: Host:kernel2ip2kernel Path:/nsm-1 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}] AwarenessGroups:[] LogLevel:INFO OpenTelemetryEndpoint:otel-collector.observability.svc.cluster.local:4317 MetricsExportInterval:10s}
Nov 24 06:58:58.253 [INFO] [cmd:[/bin/app]] sVID: "spiffe://k8s.nsm/ns/ns-kernel2ip2kernel/pod/alpine"
Nov 24 06:58:58.253 [INFO] [cmd:[/bin/app]] NSC: Connecting to Network Service Manager unix:///var/lib/networkservicemesh/nsm.io.sock
Nov 24 06:58:58.256 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:58:58.256 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:58:58.256 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:13.258 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:13.258 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:13.258 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:13.258 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:13.459 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:13.459 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:13.459 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:28.462 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:28.462 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:28.462 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:28.462 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:28.663 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:28.663 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:28.663 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:43.665 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:43.665 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:43.665 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:43.665 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:43.866 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:43.866 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:43.866 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:58.869 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:58.869 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:58.869 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:58.869 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:59.070 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:59.070 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:59.070 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:14.073 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:00:14.073 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:14.073 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:14.073 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:14.274 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:14.274 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:00:14.274 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:29.276 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:00:29.276 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:29.276 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:29.276 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:29.477 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:29.478 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:00:29.478 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:44.479 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:00:44.480 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:44.480 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:44.480 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:44.680 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:44.680 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:00:44.680 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:59.681 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:00:59.681 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:59.681 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:59.681 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:59.881 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:59.882 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:59.882 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:01:14.882 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:01:14.882 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:01:14.882 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:14.882 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:15.083 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:01:15.083 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:01:15.083 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:01:30.083 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:01:30.083 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:01:30.084 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:30.084 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:30.284 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:01:30.284 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:01:30.284 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:01:45.285 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:01:45.285 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:01:45.285 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:45.285 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:45.485 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:01:45.485 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:01:45.485 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:00.486 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:00.486 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:00.486 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:00.486 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:00.687 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:00.687 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:02:00.687 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:15.687 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:15.687 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:15.687 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:15.687 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:15.888 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:15.888 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:15.888 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:02:30.889 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:30.889 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:30.889 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:30.889 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:31.090 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:31.090 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:31.090 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:02:46.092 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:46.092 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:46.092 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:46.092 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:46.292 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:46.293 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:46.293 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:01.295 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:01.296 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:01.296 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:01.296 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:01.496 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:01.497 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:01.497 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:16.498 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:16.498 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:16.498 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:16.498 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:16.699 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:16.699 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:16.699 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:31.700 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:31.700 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:31.700 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:31.700 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:31.900 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:31.901 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:31.901 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:46.902 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:46.902 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:46.902 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:46.902 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:47.102 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:47.103 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:47.103 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:02.104 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:02.104 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:02.104 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:02.104 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:02.305 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:02.305 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:02.305 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:17.307 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:17.307 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:17.307 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:17.307 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:17.507 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:17.508 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:17.508 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:32.510 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:32.510 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:32.510 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:32.510 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:32.711 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:32.711 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:32.711 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:47.712 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:47.712 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:47.713 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:47.713 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:47.913 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:47.914 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:47.914 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:02.915 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:05:02.915 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:02.916 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:02.916 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:03.116 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:03.116 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:03.116 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:18.119 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:05:18.119 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:18.119 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:18.119 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:18.320 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:18.320 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:18.320 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:33.322 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:05:33.322 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:33.323 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:33.323 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:33.523 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:33.524 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:33.524 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:48.527 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:05:48.527 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:48.527 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:48.527 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:48.728 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:48.728 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:48.728 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:03.730 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:03.730 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:03.730 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:03.730 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:03.931 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:03.931 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:03.931 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:06:18.934 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:18.934 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:18.934 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:18.934 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:19.134 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:19.135 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:19.135 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:06:34.137 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:34.137 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:34.137 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:34.137 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:34.338 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:34.338 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:34.338 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:06:49.340 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:49.340 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:49.340 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:49.340 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:49.541 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:49.541 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:49.541 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.

My environment is roughly as follows. The namespace starting with cncp is developed by ourselves, and the others are open source.cncp should have no impact on our nsm

cncp-system            cncp-controller-manager-676f4fd966-c2x5f        1/1     Running             23 (19h ago)      10d
cncp-system            cncp-dashboard-764f84b8b6-67gc5                 1/1     Running             1 (19h ago)       10d
cncp-system            cncp-dashboard-vue-575c9d586-fbrwp              1/1     Running             1 (19h ago)       16d
cncp-system            dashboard-metrics-scraper-ddcc455b4-8vrdf       1/1     Running             1 (19h ago)       29h
cncp-system            fluentd-os-fdw44                                1/1     Running             1 (19h ago)       29h
cncp-system            fluentd-os-h7zwd                                1/1     Running             4 (19h ago)       29h
cncp-system            fluentd-os-nlxzl                                1/1     Running             3 (19h ago)       29h
cncp-system            kubernetes-dashboard-74994cc865-fn9dg           1/1     Running             1 (19h ago)       29h
cncp-system            opensearch-cluster-master-0                     1/1     Running             1 (19h ago)       29h
cncp-system            opensearch-dashboard-5d966d67b7-98hp8           1/1     Running             1 (19h ago)       29h
cncp-system            postgres-97f959c78-6n9cn                        1/1     Running             1 (19h ago)       29h
cncp-system            redis-65f88ffd9-h2zd4                           1/1     Running             1 (19h ago)       29h
cncp-system            tools-7fd7d58c8-f84bt                           1/1     Running             1 (23h ago)       29h
default                demo-dep-79c758b56d-vvj8s                       1/1     Running             1 (19h ago)       20d
default                reloader-reloader-576c66ff8c-vjdx7              1/1     Running             3 (23h ago)       30d
kube-system            cilium-8jxks                                    1/1     Running             53 (19h ago)      30d
kube-system            cilium-9bx99                                    1/1     Running             3 (19h ago)       30d
kube-system            cilium-operator-6cc4857cdd-7cjh2                1/1     Running             8 (19h ago)       30d
kube-system            cilium-zkmbk                                    1/1     Running             4 (19h ago)       30d
kube-system            coredns-6d8c4cb4d-529tv                         1/1     Running             1 (23h ago)       20d
kube-system            coredns-6d8c4cb4d-tz4k4                         1/1     Running             2 (19h ago)       21d
kube-system            etcd-cncp-ms-01                                 1/1     Running             5 (19h ago)       30d
kube-system            etcd-cncp-ms-02                                 1/1     Running             4 (19h ago)       30d
kube-system            etcd-cncp-ms-03                                 1/1     Running             5 (19h ago)       30d
kube-system            kube-apiserver-cncp-ms-01                       1/1     Running             8 (19h ago)       29d
kube-system            kube-apiserver-cncp-ms-02                       1/1     Running             4 (19h ago)       29d
kube-system            kube-apiserver-cncp-ms-03                       1/1     Running             11 (19h ago)      29d
kube-system            kube-controller-manager-cncp-ms-01              1/1     Running             7 (19h ago)       30d
kube-system            kube-controller-manager-cncp-ms-02              1/1     Running             5 (19h ago)       30d
kube-system            kube-controller-manager-cncp-ms-03              1/1     Running             6 (19h ago)       30d
kube-system            kube-multus-ds-8cmxc                            1/1     Running             3 (19h ago)       24d
kube-system            kube-multus-ds-9264m                            1/1     Running             2 (19h ago)       24d
kube-system            kube-multus-ds-jrm7c                            1/1     Running             3 (19h ago)       24d
kube-system            kube-scheduler-cncp-ms-01                       1/1     Running             8 (19h ago)       30d
kube-system            kube-scheduler-cncp-ms-02                       1/1     Running             6 (19h ago)       30d
kube-system            kube-scheduler-cncp-ms-03                       1/1     Running             6 (19h ago)       30d
kubegres-system        kubegres-controller-manager-68dcbc84-cthjc      2/2     Running             4 (19h ago)       15d
kubernetes-dashboard   dashboard-metrics-scraper-6f7c78c8cd-9bwvb      1/1     Running             3 (19h ago)       30d
kubernetes-dashboard   kubernetes-dashboard-7bb8b75596-t546d           1/1     Running             10 (19h ago)      30d
metallb-system         controller-84888b94d7-9mzsh                     1/1     Running             2 (19h ago)       22d
metallb-system         speaker-lxkm6                                   1/1     Running             80 (19h ago)      22d
metallb-system         speaker-mwfkd                                   1/1     Running             6 (19h ago)       22d
metallb-system         speaker-srr2z                                   1/1     Running             11 (19h ago)      22d
nsm-system             admission-webhook-k8s-68cbd569ff-6qfqc          1/1     Running             0                 103m
nsm-system             forwarder-vpp-2xtdd                             1/1     Running             0                 103m
nsm-system             forwarder-vpp-m8mz5                             1/1     Running             0                 103m
nsm-system             forwarder-vpp-xbmcd                             1/1     Running             0                 103m
nsm-system             nsmgr-2ksff                                     2/2     Running             0                 103m
nsm-system             nsmgr-dv9tq                                     2/2     Running             0                 103m
nsm-system             nsmgr-w8ckf                                     2/2     Running             0                 103m
nsm-system             registry-k8s-7597b84fbb-g4d82                   1/1     Running             0                 103m
openebs                openebs-localpv-provisioner-66d597678-b5t6q     1/1     Running             10 (19h ago)      30d
openebs                openebs-ndm-cluster-exporter-596d848669-65vnq   1/1     Running             3 (19h ago)       30d
openebs                openebs-ndm-hdc8v                               1/1     Running             13 (19h ago)      30d
openebs                openebs-ndm-jtvf2                               1/1     Running             8 (19h ago)       30d
openebs                openebs-ndm-node-exporter-2cttg                 1/1     Running             6 (19h ago)       30d
openebs                openebs-ndm-node-exporter-g2khr                 1/1     Running             3 (23h ago)       30d
openebs                openebs-ndm-node-exporter-r8h72                 1/1     Running             6 (19h ago)       30d
openebs                openebs-ndm-operator-8f9c7c986-lwzjn            1/1     Running             3 (23h ago)       30d
openebs                openebs-ndm-wmv55                               1/1     Running             52 (19h ago)      30d
spire                  spire-agent-28bss                               1/1     Running             0                 104m
spire                  spire-agent-928bj                               1/1     Running             0                 104m
spire                  spire-agent-96dnl                               1/1     Running             0                 104m
spire                  spire-server-0                                  2/2     Running             0                 104m

[glazychev-art]

@316953425 Thanks for the report! Could you please deploy it again and share your cluster dump?

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump
kubectl cluster-info dump -n ns-kernel2ip2kernel --output-directory nsm-dump

[316953425]

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump

hi, @glazychev-art kubectl cluster-info dump -n nsm-system --output-directory nsm-dump nsm-dump.zip kubectl cluster-info dump -n ns-kernel2ip2kernel-dual-stack --output-directory nsm-dump nsm-dump.zip kubectl cluster-info dump -n ns-kernel2ip2kernel --output-directory nsm-dump nsm-dump.zip

Is the content of the file what you want?

[glazychev-art]

Thank you. that is interesting.

Could you please change the log level from INFO to DEBUG for NSM components? https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.0/apps/nsmgr/nsmgr.yaml#L33-L34 https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.0/apps/forwarder-vpp/forwarder.yaml#L30-L31 https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.0/apps/nse-kernel/nse.yaml#L31-L32 After that, redeploy basic and Kernel2IP2Kernel

[316953425]

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump

hi，@glazychev-art kubectl cluster-info dump -n nsm-system --output-directory nsm-dump

nsm-dump.zip

kubectl cluster-info dump -n ns-kernel2ip2kernel --output-directory nsm-dump

nsm-dump.zip

[glazychev-art]

I'm sorry, my mistake. Let's please use the TRACE level

[316953425]

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump

hi @glazychev-art kubectl cluster-info dump -n nsm-system --output-directory nsm-dump nsm-dump.zip

kubectl cluster-info dump -n ns-kernel2ip2kernel --output-directory nsm-dump nsm-dump.zip

[glazychev-art]

Thanks, A few questions: Does the Kernel2Ethernet2Kernel work for you?

Is the connection between two forwarder-vpp pods fully functional? 172.16.102.12 <------> 172.16.102.13 Perhaps some packet filtering rules are used?

[316953425]

Does the Kernel2Ethernet2Kernel work for you?

Hi @glazychev-art

No packet filtering rules applyed 172.16.102.12 can ping 172.16.102.13

Kernel2Ethernet2Kernel I also tried it, alpine can start but it no interface for nsm nsm-dump.zip

[glazychev-art]

It looks like there is a problem in the connection between the pods. Perhaps there are some special parameters for cilium?

Could you please share kubectl get pods -A -o wide ?

[316953425]

kubectl get pods -A -o wide

hi, @glazychev-art yes

[root@CNCP-MS-01 use-cases]# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cncp-production dhcp4-679f4549f4-4xwd7 1/1 Running 0 2d20h 10.32.2.99 cncp-ms-02 cncp-production jool-nat-86b5f85958-szhc8 1/1 Running 1 (3d17h ago) 3d17h 10.32.1.219 cncp-ms-03 cncp-production jool-siit-55b4ccf78-q85hd 1/1 Running 1 (3d17h ago) 3d17h 10.32.1.217 cncp-ms-03 cncp-production netbox-78bc6999d7-z7s5p 2/2 Running 3 (3d17h ago) 3d20h 10.32.0.45 cncp-ms-01 cncp-production nginx-9d57d7f86-nsm7n 2/2 Running 4 (3d17h ago) 24d 10.32.0.42 cncp-ms-01 cncp-production nginx-helper-f49b8bf56-47vqg 1/1 Running 2 (3d17h ago) 24d 10.32.0.163 cncp-ms-01 cncp-system cncp-apiserver-82lrz 1/1 Running 5 (3d17h ago) 12d 172.16.102.11 cncp-ms-01 cncp-system cncp-apiserver-r25p2 1/1 Running 6 (3d17h ago) 12d 172.16.102.13 cncp-ms-03 cncp-system cncp-apiserver-wgmmt 1/1 Running 3 (3d17h ago) 12d 172.16.102.12 cncp-ms-02 cncp-system cncp-application-manager-79c9cc8f9d-7znw6 0/1 CrashLoopBackOff 984 (61s ago) 6d23h 10.32.0.150 cncp-ms-01 cncp-system cncp-controller-manager-676f4fd966-c2x5f 1/1 Running 25 (36h ago) 12d 172.16.102.12 cncp-ms-02 cncp-system cncp-dashboard-764f84b8b6-67gc5 1/1 Running 1 (3d17h ago) 12d 10.32.0.216 cncp-ms-01 cncp-system cncp-dashboard-vue-575c9d586-fbrwp 1/1 Running 1 (3d17h ago) 19d 10.32.2.250 cncp-ms-02 cncp-system dashboard-metrics-scraper-ddcc455b4-8vrdf 1/1 Running 1 (3d17h ago) 4d3h 10.32.2.52 cncp-ms-02 cncp-system fluentd-os-fdw44 1/1 Running 1 (3d17h ago) 4d3h 10.32.2.232 cncp-ms-02 cncp-system fluentd-os-h7zwd 1/1 Running 4 (3d17h ago) 4d3h 10.32.1.11 cncp-ms-03 cncp-system fluentd-os-nlxzl 1/1 Running 3 (3d17h ago) 4d3h 10.32.0.118 cncp-ms-01 cncp-system kubernetes-dashboard-74994cc865-fn9dg 1/1 Running 1 (3d17h ago) 4d3h 10.32.2.244 cncp-ms-02 cncp-system opensearch-cluster-master-0 1/1 Running 1 (3d17h ago) 4d3h 10.32.0.193 cncp-ms-01 cncp-system opensearch-dashboard-5d966d67b7-98hp8 1/1 Running 1 (3d17h ago) 4d3h 10.32.0.164 cncp-ms-01 cncp-system postgres-97f959c78-6n9cn 1/1 Running 1 (3d17h ago) 4d3h 10.32.2.53 cncp-ms-02 cncp-system redis-65f88ffd9-h2zd4 1/1 Running 1 (3d17h ago) 4d3h 10.32.0.38 cncp-ms-01 cncp-system tools-7fd7d58c8-f84bt 1/1 Running 1 (3d21h ago) 4d3h 10.32.1.135 cncp-ms-03 default demo-dep-79c758b56d-vvj8s 1/1 Running 1 (3d17h ago) 23d 10.32.0.62 cncp-ms-01 default reloader-reloader-576c66ff8c-vjdx7 1/1 Running 3 (3d21h ago) 33d 10.32.1.225 cncp-ms-03 kube-system cilium-8jxks 1/1 Running 53 (3d17h ago) 33d 172.16.102.13 cncp-ms-03 kube-system cilium-9bx99 1/1 Running 3 (3d17h ago) 33d 172.16.102.11 cncp-ms-01 kube-system cilium-operator-6cc4857cdd-7cjh2 1/1 Running 8 (3d17h ago) 33d 172.16.102.11 cncp-ms-01 kube-system cilium-zkmbk 1/1 Running 4 (3d17h ago) 33d 172.16.102.12 cncp-ms-02 kube-system coredns-6d8c4cb4d-529tv 1/1 Running 1 (3d21h ago) 23d 10.32.1.128 cncp-ms-03 kube-system coredns-6d8c4cb4d-tz4k4 1/1 Running 2 (3d17h ago) 23d 10.32.2.79 cncp-ms-02 kube-system etcd-cncp-ms-01 1/1 Running 5 (3d17h ago) 33d 172.16.102.11 cncp-ms-01 kube-system etcd-cncp-ms-02 1/1 Running 4 (3d17h ago) 33d 172.16.102.12 cncp-ms-02 kube-system etcd-cncp-ms-03 1/1 Running 5 (3d17h ago) 33d 172.16.102.13 cncp-ms-03 kube-system kube-apiserver-cncp-ms-01 1/1 Running 8 (3d17h ago) 32d 172.16.102.11 cncp-ms-01 kube-system kube-apiserver-cncp-ms-02 1/1 Running 4 (3d17h ago) 32d 172.16.102.12 cncp-ms-02 kube-system kube-apiserver-cncp-ms-03 1/1 Running 11 (3d17h ago) 32d 172.16.102.13 cncp-ms-03 kube-system kube-controller-manager-cncp-ms-01 1/1 Running 7 (3d17h ago) 33d 172.16.102.11 cncp-ms-01 kube-system kube-controller-manager-cncp-ms-02 1/1 Running 5 (3d17h ago) 33d 172.16.102.12 cncp-ms-02 kube-system kube-controller-manager-cncp-ms-03 1/1 Running 6 (3d17h ago) 33d 172.16.102.13 cncp-ms-03 kube-system kube-multus-ds-8cmxc 1/1 Running 3 (3d17h ago) 27d 172.16.102.12 cncp-ms-02 kube-system kube-multus-ds-9264m 1/1 Running 2 (3d17h ago) 27d 172.16.102.11 cncp-ms-01 kube-system kube-multus-ds-jrm7c 1/1 Running 3 (3d17h ago) 27d 172.16.102.13 cncp-ms-03 kube-system kube-scheduler-cncp-ms-01 1/1 Running 8 (3d17h ago) 33d 172.16.102.11 cncp-ms-01 kube-system kube-scheduler-cncp-ms-02 1/1 Running 6 (3d17h ago) 33d 172.16.102.12 cncp-ms-02 kube-system kube-scheduler-cncp-ms-03 1/1 Running 6 (3d17h ago) 33d 172.16.102.13 cncp-ms-03 kubegres-system kubegres-controller-manager-68dcbc84-cthjc 2/2 Running 4 (3d17h ago) 17d 10.32.1.93 cncp-ms-03 kubernetes-dashboard dashboard-metrics-scraper-6f7c78c8cd-9bwvb 1/1 Running 3 (3d17h ago) 33d 10.32.0.160 cncp-ms-01 kubernetes-dashboard kubernetes-dashboard-7bb8b75596-t546d 1/1 Running 10 (3d17h ago) 33d 10.32.0.129 cncp-ms-01 metallb-system controller-84888b94d7-9mzsh 1/1 Running 2 (3d17h ago) 25d 10.32.0.168 cncp-ms-01 metallb-system speaker-lxkm6 1/1 Running 80 (3d17h ago) 25d 172.16.102.13 cncp-ms-03 metallb-system speaker-mwfkd 1/1 Running 6 (3d17h ago) 25d 172.16.102.11 cncp-ms-01 metallb-system speaker-srr2z 1/1 Running 11 (3d17h ago) 25d 172.16.102.12 cncp-ms-02 ns-kernel2ethernet2kernel alpine 2/2 Running 0 13m 10.32.1.24 cncp-ms-03 ns-kernel2ethernet2kernel nse-kernel-56875c7d58-xnjn7 1/1 Running 0 13m 10.32.2.181 cncp-ms-02 nsm-system admission-webhook-k8s-68cbd569ff-fqs2t 1/1 Running 0 4h58m 10.32.1.148 cncp-ms-03 nsm-system forwarder-vpp-7p84f 1/1 Running 3 (4h57m ago) 4h58m 172.16.102.13 cncp-ms-03 nsm-system forwarder-vpp-8pc4x 1/1 Running 1 (4h58m ago) 4h58m 172.16.102.12 cncp-ms-02 nsm-system forwarder-vpp-lfq28 1/1 Running 2 (4h58m ago) 4h58m 172.16.102.11 cncp-ms-01 nsm-system nsmgr-2zzqg 2/2 Running 0 4h58m 10.32.2.175 cncp-ms-02 nsm-system nsmgr-k4dtp 2/2 Running 0 4h58m 10.32.1.25 cncp-ms-03 nsm-system nsmgr-x7lj7 2/2 Running 0 4h58m 10.32.0.210 cncp-ms-01 nsm-system registry-k8s-7597b84fbb-nndnr 1/1 Running 0 4h58m 10.32.1.192 cncp-ms-03 openebs openebs-localpv-provisioner-66d597678-b5t6q 1/1 Running 10 (3d17h ago) 33d 10.32.1.8 cncp-ms-03 openebs openebs-ndm-cluster-exporter-596d848669-65vnq 1/1 Running 3 (3d17h ago) 33d 10.32.0.98 cncp-ms-01 openebs openebs-ndm-hdc8v 1/1 Running 13 (3d17h ago) 33d 172.16.102.12 cncp-ms-02 openebs openebs-ndm-jtvf2 1/1 Running 8 (3d17h ago) 33d 172.16.102.11 cncp-ms-01 openebs openebs-ndm-node-exporter-2cttg 1/1 Running 6 (3d17h ago) 33d 10.32.2.188 cncp-ms-02 openebs openebs-ndm-node-exporter-g2khr 1/1 Running 3 (3d21h ago) 33d 10.32.1.169 cncp-ms-03 openebs openebs-ndm-node-exporter-r8h72 1/1 Running 6 (3d17h ago) 33d 10.32.0.149 cncp-ms-01 openebs openebs-ndm-operator-8f9c7c986-lwzjn 1/1 Running 3 (3d21h ago) 33d 10.32.1.20 cncp-ms-03 openebs openebs-ndm-wmv55 1/1 Running 52 (3d17h ago) 33d 172.16.102.13 cncp-ms-03 platform chi-my-release-clickhouse-cluster-0-0-0 0/1 Init:0/1 0 50s 10.32.2.235 cncp-ms-02 platform my-release-clickhouse-operator-6d99447df4-m7s2g 2/2 Running 2 (61s ago) 69s 10.32.2.185 cncp-ms-02 platform my-release-k8s-infra-otel-agent-5wmkn 1/1 Running 0 69s 10.32.0.64 cncp-ms-01 platform my-release-k8s-infra-otel-agent-ng2vg 1/1 Running 0 70s 10.32.2.125 cncp-ms-02 platform my-release-k8s-infra-otel-agent-rbhlb 1/1 Running 0 69s 10.32.1.236 cncp-ms-03 platform my-release-k8s-infra-otel-deployment-5f9fbc5cf4-jr7mm 1/1 Running 0 69s 10.32.2.238 cncp-ms-02 platform my-release-signoz-alertmanager-0 0/1 Init:0/1 0 68s 10.32.2.199 cncp-ms-02 platform my-release-signoz-frontend-64f9d69b5b-8f9rx 0/1 Init:0/1 0 68s 10.32.2.40 cncp-ms-02 platform my-release-signoz-otel-collector-7584d88d8b-km2hz 0/1 Init:0/1 0 69s 10.32.2.128 cncp-ms-02 platform my-release-signoz-otel-collector-metrics-586b988565-7lctf 0/1 Init:0/1 0 69s 10.32.1.237 cncp-ms-03 platform my-release-signoz-query-service-0 0/1 Init:0/1 0 69s 10.32.1.5 cncp-ms-03 platform my-release-signoz-schema-migrator-746f97548a54-scbww 0/1 Init:0/1 0 68s 10.32.1.238 cncp-ms-03 platform my-release-zookeeper-0 1/1 Running 0 68s 10.32.2.160 cncp-ms-02 spire spire-agent-28bss 1/1 Running 0 2d23h 172.16.102.13 cncp-ms-03 spire spire-agent-928bj 1/1 Running 0 2d23h 172.16.102.11 cncp-ms-01 spire spire-agent-96dnl 1/1 Running 0 2d23h 172.16.102.12 cncp-ms-02 spire spire-server-0 2/2 Running 0 2d23h 10.32.2.193 cncp-ms-02

[316953425]

hi @glazychev-art If the above one doesn't look clear, you can look at this file. env.txt

[glazychev-art]

Got it, thanks Let's check this, could you remove hostNetwork: true for forwarder and redeploy it? https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.0/apps/forwarder-vpp/forwarder.yaml#L19C1-L19C24

[316953425]

Got it, thanks Let's check this, could you remove hostNetwork: true for forwarder and redeploy it? https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.0/apps/forwarder-vpp/forwarder.yaml#L19C1-L19C24

hi @glazychev-art Also doesn't work well

The part inside the red box will cause this problem ？

[glazychev-art]

No, it is not a problem, I think

Could you share the same output from any of forwarder-vpp?

[316953425]

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump

hi @glazychev-art yes nsm-dump.zip

[glazychev-art]

looks interesting

Can you please deploy pods on different nodes and try to check availability of pod addresses? Without NSM. To check the connection you can use ping, nc for both side For example, we are interested in

1. UDP:4789
2. UDP:51820

[316953425]

hi @glazychev-art I'm not sure what you mean, are you suspecting that this address is unavailable? ​ - name: NSM_CIDR_PREFIX value: 10.234.1.1/24

[glazychev-art]

No, I suspect there are some restrictions between pods. Therefore I suggest:

1. Completely remove NSM
2. Deploy pods (for example alpine) on different nodes
3. On one pod do nc -l -u -p 51820, on the other nc -u <IP-of-the-firts-pod> 51820
4. Try to send messages

[316953425]

hi @glazychev-art @glazychev-art

it looks like work well

[316953425]

hi @glazychev-art Do you have any other suggestions? Thank you very much.

[glazychev-art]

Could you please try https://github.com/networkservicemesh/deployments-k8s/tree/v1.11.1 instead of v1.11.0 ?

[316953425]

Could you please try https://github.com/networkservicemesh/deployments-k8s/tree/v1.11.1 instead of v1.11.0 ? hi @glazychev-art I tried master, the error message is almost the same as 1.11.0 thanks for you reply

[glazychev-art]

When you tried it? The current master may not be entirely stable

[316953425]

When you tried it? The current master may not be entirely stable

ok i will try v1.11.1

[316953425]

When you tried it? The current master may not be entirely stable

hi @glazychev-art I just deployed the v1.11.1 version, but it still doesn't work. The phenomenon is the same as v1.11.0.

[glazychev-art]

Hi @316953425 , Ok, got it, thanks. Then I suggest diving into a more detailed search for the problem.

1. Set the following envs for the admission-webhook:

           - name: NSM_ENVS
             value: NSM_LOG_LEVEL=TRACE,NSM_LIVENESSCHECKENABLED=false

2. Please change the log level from INFO to TRACE for NSM components: https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.1/apps/nsmgr/nsmgr.yaml#L33-L34 https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.1/apps/forwarder-vpp/forwarder.yaml#L30-L31 https://github.com/networkservicemesh/deployments-k8s/blob/v1.11.1/apps/nse-kernel/nse.yaml#L31-L32
3. Remove hostNetwork: true for forwarder-vpp
4. Deploy NSM v1.11.1
5. Deploy Kernel2Ethernet2Kernel
6. Run ping in the separate terminal (background) NSC ---> NSE (it won't work, but still): ping -i 0.01 <NSE-IP-addr>
7. We need to find out forwarders that belong to NSC node and NSE node:

   NSC_NODE=$(kubectl get pods -l app=alpine -n ns-kernel2ethernet2kernel --template '{{range .items}}{{.spec.nodeName}}{{"\n"}}{{end}}')
   FORWARDER_NSC=$(kubectl get pods -l app=forwarder-vpp --field-selector spec.nodeName==${NSC_NODE} -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
   NSE_NODE=$(kubectl get pods -l app=nse-kernel -n ns-kernel2ethernet2kernel --template '{{range .items}}{{.spec.nodeName}}{{"\n"}}{{end}}')
   FORWARDER_NSE=$(kubectl get pods -l app=forwarder-vpp --field-selector spec.nodeName==${NSE_NODE} -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

8. Let's get more detailed information from them. I will show an example of one of them.

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl

Please attach the output of the following commands from vpp:

show int
show int addr
show hardware-interface
show errors
show acl-plugin acl
show ip neighbor
show ip fib

Then let's collect vpp traces:

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl trace add af-packet-input 1000
sleep 20 # wait a few seconds here
kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl show trace max 1000 > ${FORWARDER_NSC}_trace.log

_Please do the same but for ${FORWARDERNSE} !

9. Please show kubectl get pods -A -o wide

10. Please attach the cluster-info. You can run 2 commands at once, everything should be saved correctly in nsm-dump:

    kubectl cluster-info dump -n nsm-system --output-directory nsm-dump
    kubectl cluster-info dump -n ns-kernel2ethernet2kernel --output-directory nsm-dump

We really appreciate your help! Thanks!

[316953425]

hi @glazychev-art I seem to have found a method that you can reproduce locally. You can try it. You install k8s locally and use cilium as the cni component, and then deploy our Kernel2Ethernet2Kernel. There are no other components in my environment. Thank you very much.

My installation steps are as follows：

1、kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/spire/single_cluster?ref=v1.11.1 2、kubectl apply -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/v1.11.1/examples/spire/single_cluster/clusterspiffeid-template.yaml 3、kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/basic?ref=v1.11.1 4、kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/use-cases/Kernel2IP2Kernel?ref=v1.11.1

I installed it according to the official website. Is there any problem with my installation steps? Another thing to note is that my three nodes are both master and node at the same time. Will this have any impact?

[glazychev-art]

Thanks a lot! Your installation steps look good. Question: Is it possible to get the following output from nodes: ip a ip r ip neigh

[316953425]

hi @glazychev-art I feel that there may be a conflict between cilium and nsm, causing nsm to not be able to support functions. What do you think?

[316953425]

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl

hi @glazychev-art

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl show int show int addr show hardware-interface show errors show acl-plugin acl

Result: NSC.txt

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl trace add af-packet-input 1000 sleep 20 # wait a few seconds here kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl show trace max 1000 > ${FORWARDER_NSC}_trace.log

Result: forwarder-vpp-tm64s_trace.log

NSE Result:

NSE.txt forwarder-vpp-nr7qr_trace.log

kubectl get pods -A -o wide pod.txt

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump nsm-dump.zip

kubectl cluster-info dump -n ns-kernel2ethernet2kernel --output-directory nsm-dump

nsm-dump.zip

kubectl logs alpine -n ns-kernel2ethernet2kernel -c cmd-nsc > log cmd-nsc-log.txt

ping from nsc to nse it not work like this:

ping from nse to nsc it not work too ,like this:

I change patch-nse.yaml NSM_CIDR_PREFIX value only:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nse-kernel
spec:
  template:
    spec:
      containers:
        - name: nse
          env:
            - name: NSM_CIDR_PREFIX
              #value: 172.16.1.100/31
              value: 10.16.3.100/31
            - name: NSM_SERVICE_NAMES
              value: "kernel2ethernet2kernel"
            - name: NSM_REGISTER_SERVICE
              value: "false"

[316953425]

hi @glazychev-art If you feel that the vpp-forward information above is lacking, you can see this attachment. forwarder-vpp-tm64s_trace.log forwarder-vpp-nr7qr_trace.log

[glazychev-art]

Hi @316953425 , Thank you very much, this is very useful information!

I feel that there may be a conflict between cilium and nsm, causing nsm to not be able to support functions. What do you think?

You are right. More specifically, I think that VPP doesn't work with the cilium setup. I found a workaround locally, but there is one more thing: Still unclear why the forwarder doesn't work if hostNetwork: true is set?

If you don't mind, could you please repeat these steps, but without point 3? I slightly modified the instructions, added more VPP commands, and also indicated that ping should be started with the -i flag

Thanks! You really help a lot!

[316953425]

9. kubectl get pods -A -o wide

hi @glazychev-art， I repeat these steps and without point 3

nsc: NSC.txt forwarder-vpp-jfk7c_trace.log

nse:

NSE.txt forwarder-vpp-4wqjm_trace.log

pod: pod.txt

kubectl cluster-info dump -n nsm-system --output-directory nsm-dump nsm-dump.zip

kubectl cluster-info dump -n ns-kernel2ethernet2kernel --output-directory nsm-dump nsm-dump.zip

If you don't mind, could you tell me the local workaround you are looking for?

[glazychev-art]

Thank you, if you don’t mind, a few more traces (ping should still be running):

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl clear trace
kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl trace add virtio-input 1000
sleep 20 # wait a few seconds here
kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl show trace max 1000 > ${FORWARDER_NSC}_virtio_trace.log

[glazychev-art]

I've rebuilt the forwarder-vpp image locally, and it works for me without hostNetwork: true If you want, you can test it: artgl/cmd-forwarder-vpp:cilium_test

[316953425]

kubectl exec -it -n nsm-system ${FORWARDER_NSC} vppctl show trace max 1000 > ${FORWARDER_NSC}_virtio_trace.log

hi @glazychev-art

nsc: forwarder-vpp-jfk7c_virtio_trace.log

nse: forwarder-vpp-4wqjm_virtio_trace.log

Is this ok？ thanks

[316953425]

artgl/cmd-forwarder-vpp:cilium_test

hi @glazychev-art

I tried it locally. Whether it is Kernel2IP2Kernel or Kernel2Ethernet2Kernel, I can ping it, but the pod is always in PodInitializing.

Kernel2IP2Kernel:

Kernel2Ethernet2Kernel：

[glazychev-art]

Could you please share kubectl describe pod -n ns-kernel2ethernet2kernel alpine

[316953425]

kubectl describe pod -n ns-kernel2ethernet2kernel alpine

hi @glazychev-art , thanks apl.txt

use the artgl/cmd-forwarder-vpp:cilium_test image

[glazychev-art]

As far as I can see, the pod now has a Running state. It took some time to download the image ghcr.io/networkservicemesh/cmd-nsc:v1.11.1

[glazychev-art]

@316953425 But I think we haven't solved all the problems yet. Previously, we tried to use nс for alpine pods: https://github.com/networkservicemesh/deployments-k8s/issues/10581#issuecomment-1827211733 https://github.com/networkservicemesh/deployments-k8s/issues/10581#issuecomment-1827282829

Can we do the same thing, but on nodes? You can either go directly to the nodes (if possible), or deploy alpine pods on different nodes with hostNetwork: true and do nc

[316953425]

hi @glazychev-art

see this for alpine always in PodInitializing

kubectl describe pod -n ns-kernel2ip2kernel alpine

Name:         alpine
Namespace:    ns-kernel2ip2kernel
Priority:     0
Node:         cncp-ms-03/172.16.102.13
Start Time:   Wed, 29 Nov 2023 15:47:38 +0800
Labels:       app=alpine
              spiffe.io/spiffe-id=true
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "cilium",
                    "interface": "eth0",
                    "ips": [
                        "fd00:1032:0:1::ac1b",
                        "10.32.1.242"
                    ],
                    "mac": "de:8a:52:8a:47:82",
                    "default": true,
                    "dns": {},
                    "gateway": [
                        "fd00:1032:0:1::b494",
                        "10.32.1.224"
                    ]
                }]
              networkservicemesh.io: kernel://kernel2ip2kernel/nsm-1
Status:       Pending
IP:           10.32.1.242
IPs:
  IP:  10.32.1.242
  IP:  fd00:1032:0:1::ac1b
Init Containers:
  cmd-nsc-init:
    Container ID:   docker://ac85aae24e1084d1d00c5e1d0ec514a3fc8a35484d9b2c68be68486f7c54ec18
    Image:          ghcr.io/networkservicemesh/cmd-nsc-init:v1.11.1
    Image ID:       docker-pullable://ghcr.io/networkservicemesh/cmd-nsc-init@sha256:ab1d77fa389b0d39ba781c1acedfba141e4cdc1a72d959910056a35d682f5427
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 29 Nov 2023 15:47:40 +0800
      Finished:     Wed, 29 Nov 2023 15:47:49 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  80Mi
    Requests:
      cpu:     100m
      memory:  40Mi
    Environment:
      NSM_LOG_LEVEL:           INFO
      SPIFFE_ENDPOINT_SOCKET:  unix:///run/spire/sockets/agent.sock
      POD_NAME:                alpine (v1:metadata.name)
      NSM_NETWORK_SERVICES:    kernel://kernel2ip2kernel/nsm-1
      NSM_NAME:                $(POD_NAME)-4b24758c-d8e6-4fd2-b25a-cd175e41c082
    Mounts:
      /run/spire/sockets from spire-agent-socket (ro)
      /var/lib/networkservicemesh from nsm-socket (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2hv7h (ro)
Containers:
  alpine:
    Container ID:
    Image:         alpine:3.15.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2hv7h (ro)
  cmd-nsc:
    Container ID:
    Image:          ghcr.io/networkservicemesh/cmd-nsc:v1.11.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  80Mi
    Requests:
      cpu:     100m
      memory:  40Mi
    Environment:
      NSM_LOG_LEVEL:           INFO
      SPIFFE_ENDPOINT_SOCKET:  unix:///run/spire/sockets/agent.sock
      POD_NAME:                alpine (v1:metadata.name)
      NSM_NETWORK_SERVICES:    kernel://kernel2ip2kernel/nsm-1
      NSM_NAME:                $(POD_NAME)-4b24758c-d8e6-4fd2-b25a-cd175e41c082
    Mounts:
      /run/spire/sockets from spire-agent-socket (ro)
      /var/lib/networkservicemesh from nsm-socket (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2hv7h (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-2hv7h:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  spire-agent-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /run/spire/sockets
    HostPathType:  Directory
  nsm-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/networkservicemesh
    HostPathType:  Directory
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       67s   default-scheduler  Successfully assigned ns-kernel2ip2kernel/alpine to cncp-ms-03
  Normal  AddedInterface  66s   multus             Add eth0 [fd00:1032:0:1::ac1b/128 10.32.1.242/32] from cilium
  Normal  Pulled          66s   kubelet            Container image "ghcr.io/networkservicemesh/cmd-nsc-init:v1.11.1" already present on machine
  Normal  Created         65s   kubelet            Created container cmd-nsc-init
  Normal  Started         65s   kubelet            Started container cmd-nsc-init
  Normal  Pulled          55s   kubelet            Container image "alpine:3.15.0" already present on machine
  Normal  Created         55s   kubelet            Created container alpine
  Normal  Started         55s   kubelet            Started container alpine
  Normal  Pulling         55s   kubelet            Pulling image "ghcr.io/networkservicemesh/cmd-nsc:v1.11.1"

kubectl describe pod -n ns-kernel2ethernet2kernel alpine

Name:         alpine
Namespace:    ns-kernel2ethernet2kernel
Priority:     0
Node:         cncp-ms-03/172.16.102.13
Start Time:   Wed, 29 Nov 2023 15:51:26 +0800
Labels:       app=alpine
              spiffe.io/spiffe-id=true
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "cilium",
                    "interface": "eth0",
                    "ips": [
                        "fd00:1032:0:1::4dae",
                        "10.32.1.222"
                    ],
                    "mac": "d2:af:f5:c2:3d:2d",
                    "default": true,
                    "dns": {},
                    "gateway": [
                        "fd00:1032:0:1::b494",
                        "10.32.1.224"
                    ]
                }]
              networkservicemesh.io: kernel://kernel2ethernet2kernel/nsm-1
Status:       Pending
IP:           10.32.1.222
IPs:
  IP:  10.32.1.222
  IP:  fd00:1032:0:1::4dae
Init Containers:
  cmd-nsc-init:
    Container ID:   docker://83aad383e15915fc64cd4e113c7ec021d08df014623574b367b427f9ce497698
    Image:          ghcr.io/networkservicemesh/cmd-nsc-init:v1.11.1
    Image ID:       docker-pullable://ghcr.io/networkservicemesh/cmd-nsc-init@sha256:ab1d77fa389b0d39ba781c1acedfba141e4cdc1a72d959910056a35d682f5427
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 29 Nov 2023 15:51:28 +0800
      Finished:     Wed, 29 Nov 2023 15:51:36 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  80Mi
    Requests:
      cpu:     100m
      memory:  40Mi
    Environment:
      NSM_LOG_LEVEL:           INFO
      SPIFFE_ENDPOINT_SOCKET:  unix:///run/spire/sockets/agent.sock
      POD_NAME:                alpine (v1:metadata.name)
      NSM_NETWORK_SERVICES:    kernel://kernel2ethernet2kernel/nsm-1
      NSM_NAME:                $(POD_NAME)-f58167f0-fbc9-4673-b2be-41815d911c22
    Mounts:
      /run/spire/sockets from spire-agent-socket (ro)
      /var/lib/networkservicemesh from nsm-socket (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gdbgh (ro)
Containers:
  alpine:
    Container ID:
    Image:         alpine:3.15.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gdbgh (ro)
  cmd-nsc:
    Container ID:
    Image:          ghcr.io/networkservicemesh/cmd-nsc:v1.11.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  80Mi
    Requests:
      cpu:     100m
      memory:  40Mi
    Environment:
      NSM_LOG_LEVEL:           INFO
      SPIFFE_ENDPOINT_SOCKET:  unix:///run/spire/sockets/agent.sock
      POD_NAME:                alpine (v1:metadata.name)
      NSM_NETWORK_SERVICES:    kernel://kernel2ethernet2kernel/nsm-1
      NSM_NAME:                $(POD_NAME)-f58167f0-fbc9-4673-b2be-41815d911c22
    Mounts:
      /run/spire/sockets from spire-agent-socket (ro)
      /var/lib/networkservicemesh from nsm-socket (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gdbgh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-gdbgh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  spire-agent-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /run/spire/sockets
    HostPathType:  Directory
  nsm-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/networkservicemesh
    HostPathType:  Directory
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       49s   default-scheduler  Successfully assigned ns-kernel2ethernet2kernel/alpine to cncp-ms-03
  Normal  AddedInterface  48s   multus             Add eth0 [fd00:1032:0:1::4dae/128 10.32.1.222/32] from cilium
  Normal  Pulled          48s   kubelet            Container image "ghcr.io/networkservicemesh/cmd-nsc-init:v1.11.1" already present on machine
  Normal  Created         47s   kubelet            Created container cmd-nsc-init
  Normal  Started         47s   kubelet            Started container cmd-nsc-init
  Normal  Pulled          38s   kubelet            Container image "alpine:3.15.0" already present on machine
  Normal  Created         38s   kubelet            Created container alpine
  Normal  Started         37s   kubelet            Started container alpine
  Normal  Pulling         37s   kubelet            Pulling image "ghcr.io/networkservicemesh/cmd-nsc:v1.11.1"

[316953425]

Can we do the same thing, but on nodes? You can either go directly to the nodes (if possible), or deploy alpine pods on different nodes with hostNetwork: true and do nc

hi @glazychev-art

I understand what you mean： 1、deploy nsm with hostNetwork: true 2、In one node exec : nc -l -u -p 51820 3、In another node exec ：nc -u 51820 and, and send messages

like this:

172.16.102.11 is the node ip

Am I understanding correctly?

[316953425]

As far as I can see, the pod now has a Running state. It took some time to download the image ghcr.io/networkservicemesh/cmd-nsc:v1.11.1

hi @glazychev-art After a long time ,alpine automatically changed to running, I didn’t do anything I think this time is a bit long, maybe there are some problems

[glazychev-art]

172.16.102.11 is the node ip

Am I understanding correctly?

Yes, you are. This looks very strange. Question: does Kernel2IP2Kernel work with the image artgl/cmd-forwarder-vpp:cilium_test and hostNetwork: true for forwarder?

After a long time ,alpine automatically changed to running, I didn’t do anything I think this time is a bit long, maybe there are some problems

There is no problem here, downloading the docker image may take some time

[316953425]

172.16.102.11 is the node ip Am I understanding correctly?

Yes, you are. This looks very strange. Question: does Kernel2IP2Kernel work with the image artgl/cmd-forwarder-vpp:cilium_test and hostNetwork: true for forwarder?

After a long time ,alpine automatically changed to running, I didn’t do anything I think this time is a bit long, maybe there are some problems

There is no problem here, downloading the docker image may take some time

hi @glazychev-art

Kernel2IP2Kernel work with the image artgl/cmd-forwarder-vpp:cilium_test but remove hostNetwork: true for forwarder

If with hostNetwork: true for forwarder spire agent and server can not start

[glazychev-art]

If with hostNetwork: true for forwarder spire agent and server can not start

This shouldn't happen I think. Could you try to redeploy everything (including spire), but wait until spire-agents and spire-server are ready before NSM deploying

[316953425]

If with hostNetwork: true for forwarder spire agent and server can not start

This shouldn't happen I think. Could you try to redeploy everything (including spire), but wait until spire-agents and spire-server are ready before NSM deploying

hi @glazychev-art with the image artgl/cmd-forwarder-vpp:cilium_test with hostNetwork: true

deploy spire delploy nsm delploy kernel2ethernet2kernel

is right? cat apps/forwarder-vpp/forwarder.yaml

spire agent and server can not start

[316953425]

hi @glazychev-art with the image artgl/cmd-forwarder-vpp:cilium_test not with hostNetwork: true

deploy spire delploy nsm delploy kernel2ethernet2kernel

spire agent and server work well

[glazychev-art]

Try just deploying spire. Don't deploy NSM yet. Wait until spire is Running