Kernel2IP2Kernel alpine pod cannot start in production environment

[316953425]

hi @glazychev-art and other developers

I really like the nsm project, it feels like an exciting project and I'm in production this week， I deploy deployments-k8s version 1.11.0, and then deploy the officially provided Kernel2IP2Kernel example, but the alpine pod cannot be started. The error message is as follows:

[root@CNCP-MS-01 use-cases]# kubectl logs -f alpine -n ns-kernel2ip2kernel -c cmd-nsc-init
time="2023-11-24T06:58:54Z" level=info msg="Starting NetworkServiceMesh Client ..."
This application is configured via the environment. The following environment
variables can be used:

KEY                            TYPE                                                   DEFAULT                                                REQUIRED    DESCRIPTION
NSM_NAME                       String                                                 cmd-nsc-init                                                       Name of the client
NSM_DIAL_TIMEOUT               Duration                                               5s                                                                 timeout to dial NSMgr
NSM_REQUEST_TIMEOUT            Duration                                               15s                                                                timeout to request NSE
NSM_CONNECT_TO                 URL                                                    unix:///var/lib/networkservicemesh/nsm.io.sock                     url to connect to
NSM_MAX_TOKEN_LIFETIME         Duration                                               10m                                                                maximum lifetime of tokens
NSM_NETWORK_SERVICES           Comma-separated list of URL                                                                                               A list of Network Service Requests
NSM_AWARENESS_GROUPS           Comma-separated list of Comma-separated list of URL                                                                       Awareness groups for mutually aware NSEs
NSM_LOG_LEVEL                  String                                                 INFO                                                               Log level
NSM_OPENTELEMETRYENDPOINT      String                                                 otel-collector.observability.svc.cluster.local:4317                OpenTelemetry Collector Endpoint
NSM_METRICS_EXPORT_INTERVAL    Duration                                               10s                                                                interval between mertics exports
Nov 24 06:58:54.600 [INFO] [cmd:[/bin/app]] rootConf: &{Name:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4 DialTimeout:5s RequestTimeout:15s ConnectTo:{Scheme:unix Opaque: User: Host: Path:/var/lib/networkservicemesh/nsm.io.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:} MaxTokenLifetime:10m0s NetworkServices:[{Scheme:kernel Opaque: User: Host:kernel2ip2kernel Path:/nsm-1 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}] AwarenessGroups:[] LogLevel:INFO OpenTelemetryEndpoint:otel-collector.observability.svc.cluster.local:4317 MetricsExportInterval:10s}
Nov 24 06:58:58.253 [INFO] [cmd:[/bin/app]] sVID: "spiffe://k8s.nsm/ns/ns-kernel2ip2kernel/pod/alpine"
Nov 24 06:58:58.253 [INFO] [cmd:[/bin/app]] NSC: Connecting to Network Service Manager unix:///var/lib/networkservicemesh/nsm.io.sock
Nov 24 06:58:58.256 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:58:58.256 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:58:58.256 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:13.258 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:13.258 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:13.258 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:13.258 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:13.459 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:13.459 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:13.459 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:28.462 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:28.462 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:28.462 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:28.462 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:28.663 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:28.663 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:28.663 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:43.665 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:43.665 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:43.665 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:43.665 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:43.866 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:43.866 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 06:59:43.866 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:58.869 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 06:59:58.869 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 06:59:58.869 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:58.869 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 06:59:59.070 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 06:59:59.070 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 06:59:59.070 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:14.073 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:00:14.073 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:14.073 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:14.073 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:14.274 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:14.274 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:00:14.274 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:29.276 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:00:29.276 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:29.276 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:29.276 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:00:29.477 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:29.478 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:00:29.478 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:44.479 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:00:44.480 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:44.480 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:44.480 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:44.680 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:44.680 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:00:44.680 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:59.681 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:00:59.681 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:00:59.681 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:59.681 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:00:59.881 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:00:59.882 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:00:59.882 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:01:14.882 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:01:14.882 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:01:14.882 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:14.882 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:15.083 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:01:15.083 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:01:15.083 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:01:30.083 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:01:30.083 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:01:30.084 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:30.084 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:30.284 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:01:30.284 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:01:30.284 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:01:45.285 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:01:45.285 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:01:45.285 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:45.285 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:01:45.485 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:01:45.485 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:01:45.485 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:00.486 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:00.486 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:00.486 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:00.486 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:00.687 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:00.687 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:02:00.687 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:15.687 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:15.687 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:15.687 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:15.687 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:15.888 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:15.888 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:15.888 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:02:30.889 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:30.889 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:30.889 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:30.889 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:31.090 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:31.090 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:31.090 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:02:46.092 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:02:46.092 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:02:46.092 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:46.092 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:02:46.292 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:02:46.293 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:02:46.293 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:01.295 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:01.296 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:01.296 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:01.296 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:01.496 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:01.497 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:01.497 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:16.498 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:16.498 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:16.498 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:16.498 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:16.699 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:16.699 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:16.699 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:31.700 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:31.700 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:31.700 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:31.700 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:31.900 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:31.901 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:31.901 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:03:46.902 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:03:46.902 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:03:46.902 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:46.902 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:03:47.102 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:03:47.103 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:03:47.103 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:02.104 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:02.104 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:02.104 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:02.104 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:02.305 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:02.305 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:02.305 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:17.307 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:17.307 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:17.307 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:17.307 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:17.507 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:17.508 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:17.508 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:32.510 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:32.510 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:32.510 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:32.510 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:32.711 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:32.711 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:32.711 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:04:47.712 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:04:47.712 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:04:47.713 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:47.713 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:04:47.913 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:04:47.914 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:04:47.914 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:02.915 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:05:02.915 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:02.916 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:02.916 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:03.116 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:03.116 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:03.116 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:18.119 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 24 07:05:18.119 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:18.119 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:18.119 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
Nov 24 07:05:18.320 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:18.320 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:18.320 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:33.322 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:05:33.322 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:33.323 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:33.323 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:33.523 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:33.524 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:33.524 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:05:48.527 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:05:48.527 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:05:48.527 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:48.527 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:05:48.728 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:05:48.728 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:05:48.728 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:03.730 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:03.730 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:03.730 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:03.730 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:03.931 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:03.931 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:03.931 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:06:18.934 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:18.934 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:18.934 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:18.934 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:19.134 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:19.135 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:19.135 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:06:34.137 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:34.137 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:34.137 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:34.137 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:34.338 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:34.338 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:34.338 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.
Nov 24 07:06:49.340 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
Nov 24 07:06:49.340 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request-response=null
Nov 24 07:06:49.340 [ERRO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Error returned from sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:49.340 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-2xtdd --> rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
Nov 24 07:06:49.541 [INFO] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] client-request={"connection":{"id":"alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
Nov 24 07:06:49.541 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable CLUSTER_NAME is not set. Skipping.
Nov 24 07:06:49.541 [WARN] [id:alpine-e1070754-a6d7-44da-b404-002f7bcdc3a4-0] [type:networkService] Environment variable NODE_NAME is not set. Skipping.

My environment is roughly as follows. The namespace starting with cncp is developed by ourselves, and the others are open source.cncp should have no impact on our nsm

cncp-system            cncp-controller-manager-676f4fd966-c2x5f        1/1     Running             23 (19h ago)      10d
cncp-system            cncp-dashboard-764f84b8b6-67gc5                 1/1     Running             1 (19h ago)       10d
cncp-system            cncp-dashboard-vue-575c9d586-fbrwp              1/1     Running             1 (19h ago)       16d
cncp-system            dashboard-metrics-scraper-ddcc455b4-8vrdf       1/1     Running             1 (19h ago)       29h
cncp-system            fluentd-os-fdw44                                1/1     Running             1 (19h ago)       29h
cncp-system            fluentd-os-h7zwd                                1/1     Running             4 (19h ago)       29h
cncp-system            fluentd-os-nlxzl                                1/1     Running             3 (19h ago)       29h
cncp-system            kubernetes-dashboard-74994cc865-fn9dg           1/1     Running             1 (19h ago)       29h
cncp-system            opensearch-cluster-master-0                     1/1     Running             1 (19h ago)       29h
cncp-system            opensearch-dashboard-5d966d67b7-98hp8           1/1     Running             1 (19h ago)       29h
cncp-system            postgres-97f959c78-6n9cn                        1/1     Running             1 (19h ago)       29h
cncp-system            redis-65f88ffd9-h2zd4                           1/1     Running             1 (19h ago)       29h
cncp-system            tools-7fd7d58c8-f84bt                           1/1     Running             1 (23h ago)       29h
default                demo-dep-79c758b56d-vvj8s                       1/1     Running             1 (19h ago)       20d
default                reloader-reloader-576c66ff8c-vjdx7              1/1     Running             3 (23h ago)       30d
kube-system            cilium-8jxks                                    1/1     Running             53 (19h ago)      30d
kube-system            cilium-9bx99                                    1/1     Running             3 (19h ago)       30d
kube-system            cilium-operator-6cc4857cdd-7cjh2                1/1     Running             8 (19h ago)       30d
kube-system            cilium-zkmbk                                    1/1     Running             4 (19h ago)       30d
kube-system            coredns-6d8c4cb4d-529tv                         1/1     Running             1 (23h ago)       20d
kube-system            coredns-6d8c4cb4d-tz4k4                         1/1     Running             2 (19h ago)       21d
kube-system            etcd-cncp-ms-01                                 1/1     Running             5 (19h ago)       30d
kube-system            etcd-cncp-ms-02                                 1/1     Running             4 (19h ago)       30d
kube-system            etcd-cncp-ms-03                                 1/1     Running             5 (19h ago)       30d
kube-system            kube-apiserver-cncp-ms-01                       1/1     Running             8 (19h ago)       29d
kube-system            kube-apiserver-cncp-ms-02                       1/1     Running             4 (19h ago)       29d
kube-system            kube-apiserver-cncp-ms-03                       1/1     Running             11 (19h ago)      29d
kube-system            kube-controller-manager-cncp-ms-01              1/1     Running             7 (19h ago)       30d
kube-system            kube-controller-manager-cncp-ms-02              1/1     Running             5 (19h ago)       30d
kube-system            kube-controller-manager-cncp-ms-03              1/1     Running             6 (19h ago)       30d
kube-system            kube-multus-ds-8cmxc                            1/1     Running             3 (19h ago)       24d
kube-system            kube-multus-ds-9264m                            1/1     Running             2 (19h ago)       24d
kube-system            kube-multus-ds-jrm7c                            1/1     Running             3 (19h ago)       24d
kube-system            kube-scheduler-cncp-ms-01                       1/1     Running             8 (19h ago)       30d
kube-system            kube-scheduler-cncp-ms-02                       1/1     Running             6 (19h ago)       30d
kube-system            kube-scheduler-cncp-ms-03                       1/1     Running             6 (19h ago)       30d
kubegres-system        kubegres-controller-manager-68dcbc84-cthjc      2/2     Running             4 (19h ago)       15d
kubernetes-dashboard   dashboard-metrics-scraper-6f7c78c8cd-9bwvb      1/1     Running             3 (19h ago)       30d
kubernetes-dashboard   kubernetes-dashboard-7bb8b75596-t546d           1/1     Running             10 (19h ago)      30d
metallb-system         controller-84888b94d7-9mzsh                     1/1     Running             2 (19h ago)       22d
metallb-system         speaker-lxkm6                                   1/1     Running             80 (19h ago)      22d
metallb-system         speaker-mwfkd                                   1/1     Running             6 (19h ago)       22d
metallb-system         speaker-srr2z                                   1/1     Running             11 (19h ago)      22d
nsm-system             admission-webhook-k8s-68cbd569ff-6qfqc          1/1     Running             0                 103m
nsm-system             forwarder-vpp-2xtdd                             1/1     Running             0                 103m
nsm-system             forwarder-vpp-m8mz5                             1/1     Running             0                 103m
nsm-system             forwarder-vpp-xbmcd                             1/1     Running             0                 103m
nsm-system             nsmgr-2ksff                                     2/2     Running             0                 103m
nsm-system             nsmgr-dv9tq                                     2/2     Running             0                 103m
nsm-system             nsmgr-w8ckf                                     2/2     Running             0                 103m
nsm-system             registry-k8s-7597b84fbb-g4d82                   1/1     Running             0                 103m
openebs                openebs-localpv-provisioner-66d597678-b5t6q     1/1     Running             10 (19h ago)      30d
openebs                openebs-ndm-cluster-exporter-596d848669-65vnq   1/1     Running             3 (19h ago)       30d
openebs                openebs-ndm-hdc8v                               1/1     Running             13 (19h ago)      30d
openebs                openebs-ndm-jtvf2                               1/1     Running             8 (19h ago)       30d
openebs                openebs-ndm-node-exporter-2cttg                 1/1     Running             6 (19h ago)       30d
openebs                openebs-ndm-node-exporter-g2khr                 1/1     Running             3 (23h ago)       30d
openebs                openebs-ndm-node-exporter-r8h72                 1/1     Running             6 (19h ago)       30d
openebs                openebs-ndm-operator-8f9c7c986-lwzjn            1/1     Running             3 (23h ago)       30d
openebs                openebs-ndm-wmv55                               1/1     Running             52 (19h ago)      30d
spire                  spire-agent-28bss                               1/1     Running             0                 104m
spire                  spire-agent-928bj                               1/1     Running             0                 104m
spire                  spire-agent-96dnl                               1/1     Running             0                 104m
spire                  spire-server-0                                  2/2     Running             0                 104m

[316953425]

Try just deploying spire. Don't deploy NSM yet. Wait until spire is Running

hi @glazychev-art

my env already deploy spire nsm and kernel2ethernet2kernel

current forward.yam is :

with the image artgl/cmd-forwarder-vpp:cilium_test not with hostNetwork: true

I'm a little unsure Do you mean to directly redeploy spire? No need to redeploy nsm and kernel2ethernet2kernel. ​ Does forward.yaml need to use cmd-forwarder-vpp:cilium_test and hostNetwork: true?

thanks

[glazychev-art]

1. Please delete:
   • kernel2ethernet2kernel
   • NSM
   • spire
2. Redeploy spire again and wait Running state for agents and server

[316953425]

1. Please delete:

• kernel2ethernet2kernel
• NSM
• spire

2. Redeploy spire again and wait Running state for agents and server

hi @glazychev-art with the image artgl/cmd-forwarder-vpp:cilium_test with hostNetwork: true

i do it

[glazychev-art]

At the moment it doesn't matter to us

with the image artgl/cmd-forwarder-vpp:cilium_test with hostNetwork: true

because we haven't deployed NSM yet. It's right?

Could you please share kubectl describe ds -n spire

[316953425]

kubectl describe ds -n spire hi @glazychev-art

[root@CNCP-MS-01 deployments-k8s-release-v1.11.1]# kubectl describe ds -n spire
Name:           spire-agent
Selector:       app=spire-agent
Node-Selector:  <none>
Labels:         app=spire-agent
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels:           app=spire-agent
Service Account:  spire-agent
Init Containers:
init:
Image:      gcr.io/spiffe-io/wait-for-it
Port:       <none>
Host Port:  <none>
Args:
-t
30
spire-server:8081
Environment:  <none>
Mounts:       <none>
init-bundle:
Image:      gcr.io/spiffe-io/wait-for-it
Port:       <none>
Host Port:  <none>
Command:
sh
-c
t=0; until [ -f /run/spire/bundle/bundle.crt 2>&1 ] || [ $t -eq 60 ]; do t=`expr $t + 1`; sleep 1; done
Environment:  <none>
Mounts:
/run/spire/bundle from spire-bundle (rw)
Containers:
spire-agent:
Image:      ghcr.io/spiffe/spire-agent:1.6.1
Port:       <none>
Host Port:  <none>
Args:
-config
/run/spire/config/agent.conf
Liveness:     exec [/opt/spire/bin/spire-agent healthcheck -socketPath /run/spire/sockets/agent.sock] delay=15s timeout=3s period=60s #success=1 #failure=2
Readiness:    exec [/opt/spire/bin/spire-agent healthcheck -socketPath /run/spire/sockets/agent.sock --shallow] delay=5s timeout=1s period=5s #success=1 #failure=3
Environment:  <none>
Mounts:
/run/spire/bundle from spire-bundle (rw)
/run/spire/config from spire-config (ro)
/run/spire/sockets from spire-agent-socket (rw)
/var/run/secrets/tokens from spire-token (rw)
Volumes:
spire-config:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      spire-agent
Optional:  false
spire-bundle:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      spire-bundle
Optional:  false
spire-agent-socket:
Type:          HostPath (bare host directory volume)
Path:          /run/spire/sockets
HostPathType:  DirectoryOrCreate
spire-token:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  7200
Events:
Type    Reason            Age    From                  Message
----    ------            ----   ----                  -------
Normal  SuccessfulCreate  4m40s  daemonset-controller  Created pod: spire-agent-6cwrq
Normal  SuccessfulCreate  4m39s  daemonset-controller  Created pod: spire-agent-hf6lv
Normal  SuccessfulCreate  4m39s  daemonset-controller  Created pod: spire-agent-84rx8

[glazychev-art]

Thanks! Please don't forget for spire kubectl apply -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/v1.11.1/examples/spire/single_cluster/clusterspiffeid-template.yaml

Now, please deploy NSM with the image artgl/cmd-forwarder-vpp:cilium_test and hostNetwork: true

Is spire still Running?

[316953425]

because we haven't deployed NSM yet. It's right?

hi @glazychev-art
yes spire is running

with the image artgl/cmd-forwarder-vpp:cilium_test and hostNetwork: true

deploy NSM done

[glazychev-art]

Ok, cool Please, deploy Kernel2Ethernet2Kernel

[316953425]

Ok, cool Please, deploy Kernel2Ethernet2Kernel

hi @glazychev-art done

alpine start but nsm-1 periodically created and deleted

like this :

[glazychev-art]

Okay, got it, thank you very much! This is strange to me.

In that case, it would be great if you could share your environment. Can I somehow create the same cluster as yours?

[316953425]

Okay, got it, thank you very much! This is strange to me.

In that case, it would be great if you could share your environment. Can I somehow create the same cluster as yours?

hi @glazychev-art I think if you are in need, I can try my best to help you, as long as I know how to deal with it

my cilium file no others in k8s cluster cilium.txt Change the suffix to yaml

[glazychev-art]

No, I mean k8s-cluster. You sent me a cilium CNI config.

Which k8s provider are you using? For example, AWS, AKS, GKE?

[316953425]

No, I mean k8s-cluster. You sent me a cilium CNI config.

Which k8s provider are you using? For example, AWS, AKS, GKE?

hi @glazychev-art I installed the k8s cluster locally and used three physical servers. I did not use k8s provided by the cloud service provider.

The core configuration is as follows： 1、linux kernel：

    - kernel-4.19.94-300.el7.x86_64.rpm
    - kernel-core-4.19.94-300.el7.x86_64.rpm
    - kernel-devel-4.19.94-300.el7.x86_64.rpm
    - kernel-headers-4.19.94-300.el7.x86_64.rpm
    - kernel-modules-4.19.94-300.el7.x86_64.rpm

2、systemctl disable firewalld && systemctl stop firewalld systemctl disable NetworkManager && systemctl stop NetworkManager

3、selinux /etc/selinux/config：SELINUX=disabled

4、swap echo "vm.swappiness=0" >> /etc/sysctl.conf sed -i 's$/dev/mapper/centos-swap$#/dev/mapper/centos-swap$g' /etc/fstab

5、sysctl.conf

net.ipv6.conf.eth0.accept_dad = 0 net.ipv6.conf.eth0.accept_ra = 1 net.ipv6.conf.eth0.accept_ra_defrtr = 1 net.ipv6.conf.eth0.accept_ra_rtr_pref = 1 net.ipv6.conf.eth0.accept_ra_rt_info_max_plen = 1

6、time yum install -y chrony sed -i.bak '3,6d' /etc/chrony.conf && sed -i '3c server ntp1.aliyun.com iburst' /etc/chrony.conf systemctl enable chronyd --now && systemctl restart chronyd chronyc sources mv /etc/localtime /etc/localtime.bak && ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

7、/etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 modprobe br_netfilter && sysctl -p /etc/sysctl.d/k8s.conf 8、ipvs

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

!/bin/bash

modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh modprobe nf_conntrack EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules yum -y install ipset ipvsadm 9、 yum install -y ebtables socat ipset conntrack yum install -y yum-utils yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y docker-ce systemctl enable docker && systemctl restart docker

sed -i.bak "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service systemctl daemon-reload && systemctl restart docker && systemctl status docker 10、kubernetes

cat > /etc/yum.repos.d/kubernetes.repo <<EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF

yum makecache fast

11、yum install -y kubelet-1.23.10 kubectl-1.23.10 kubeadm-1.23.10 systemctl enable kubelet

[glazychev-art]

Thank you very much for the script, but unfortunately I cannot reproduce your problem.

I can suggest the following:

1. Follow this script, but please skip steps 3, 7, 8, 9, 10

2. On the node where NSE is deployed, please run tcpdump. We are interested in frames with dst: (for example, 172.16.102.11) port 4789

[316953425]

2. On the node where NSE is deployed, please run tcpdump. We are interested in frames with dst: (for example, 172.16.102.11) port 4789

hi @glazychev-art see it a.cap.txt See if there is the information you want here. Change the file suffix.

[glazychev-art]

Yes, thank you. As I can see, the required traffic does not arrive at the NSE node.

Can we collect the same but from the NSC node?

[316953425]

Yes, thank you. As I can see, the required traffic does not arrive at the NSE node.

Can we collect the same but from the NSC node?

HI @glazychev-art

no package too

[glazychev-art]

@316953425 Actually we are interested in UDP Then can we collect frames from NSE node again?

[316953425]

@316953425 Actually we are interested in UDP Then can we collect frames from NSE node again?

hi @glazychev-art

nsc node:

nse node:

I have posted the detailed information. Can you help me see if it is correct? UDP also has no data packets, whether in nsc-node or nse-node. thanks

[glazychev-art]

This tcpdump flag doesn't help either? tcpdump -i ens3 udp port 4789 --immediate-mode

[316953425]

tcpdump -i ens3 udp port 4789 --immediate-mode

hi @glazychev-art no package

[glazychev-art]

Ok, thanks Correct me if I'm wrong: this example worked for you on NSM v1.6.1, right?

[316953425]

Correct me if I'm wrong: this example worked for you on NSM v1.6.1, right? hi @glazychev-art v1.6.1 v1.11.0 v1.11.1 master all not working properly

[316953425]

hi @glazychev-art I have built three environments, but none of them work if I use cilium. Have you reproduced using cilium locally?

[glazychev-art]

Got it. So far I'm having problems with cilium on my servers. I was only able to reproduce the problem without hostNetwork: true, so we fixed it.

Question: have you tried other CNI plugins? For example flannel? https://github.com/flannel-io/flannel

[316953425]

Got it. So far I'm having problems with cilium on my servers. I was only able to reproduce the problem without hostNetwork: true, so we fixed it.

Question: have you tried other CNI plugins? For example flannel? https://github.com/flannel-io/flannel

hi @glazychev-art i not try other cni

[glazychev-art]

It would be great if you could check another CNI. This way we could determine whether the problem is really in the cilium CNI plugin or somewhere in the configuration of your servers.

[316953425]

Got it. So far I'm having problems with cilium on my servers. I was only able to reproduce the problem without hostNetwork: true, so we fixed it.

hi @glazychev-art

On the server you use cilium, is there any problem if you use hostNetwork: true?

[316953425]

hi @glazychev-art I found that the ip prefix of the nsm-1 interface of nsc and nse is /32. How can I let nsc or nse get the ip prefix of the interface to be /24?

[glazychev-art]

I found that the ip prefix of the nsm-1 interface of nsc and nse is /32. How can I let nsc or nse get the ip prefix of the interface to be /24?

This is a separate topic. In short, NSM allocates /32 addresses for P2P connections. You can look at these chain elements in more detail: https://github.com/networkservicemesh/cmd-nse-icmp-responder/blob/main/main.go#L208 https://github.com/networkservicemesh/sdk/tree/main/pkg/networkservice/ipam/groupipam https://github.com/networkservicemesh/sdk/tree/main/pkg/networkservice/ipam/point2pointipam

On the server you use cilium, is there any problem if you use hostNetwork: true?

On my local cluster, cilium works fine with hostNetwork: true. But maybe my configuration is different. I believe that the problem may also be in the configuration of your servers. So to find out, can you please remove cilium and try other CNI plugin (eg flannel)?

[316953425]

hi @glazychev-art I tried calico, no problems， it work well thanks

env：

Do you need my cilium configuration？

[glazychev-art]

Yes, please

[316953425]

Yes, please

hi @glazychev-art
see it cilium.yaml.txt

[glazychev-art]

So, I've tested cilium on standalone servers with Ubuntu 20.04 on which kubernetes is installed, using this cilium guide: https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/ cilium install --version 1.14.4 And it works well for me with hostNetwork: true on forwarders.

Edit: works on CentOS 7 too

Perhaps checking your cilium installation will help you: https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#validate-the-installation

[316953425]

cilium

hi @glazychev-art Can you send me a copy of your cilium configuration so that I can check the difference with my configuration? If you don't mind, you can send it to my email (316953425@qq.com) thanks a lot

[glazychev-art]

I used only cilium install --version 1.14.4, without any changes Config (I removed a few Secrets from there): cilium_config.txt

[denis-tingaikin]

It seems like this one is fixed by https://github.com/networkservicemesh/cmd-forwarder-vpp/pull/1003.

If it's still reproducing, feel free to reopen.

(Also, I've renamed the ticket from 'Urgent' to 'ASAP' label because we want to add this to the release notes)