search

networkservicemesh / sdk

Apache License 2.0
34 stars 36 forks source link

connectServer Request fails with context canceled error during refresh period #1068

Open pperiyasamy opened 3 years ago

pperiyasamy commented 3 years ago

Expected Behavior

The refresh or healing should work seamlessly when endpoint and client are connected and traffic is running fine among those pods.

Current Behavior

The forwarder's connectServer.Request() fails with context canceled error for unknown reason, this would cause timeout handler to invoke Closeof forwarder which would teardown the connection between client and endpoint.

Failure Information (for bugs)

Steps to Reproduce

  1. Bring up NSC and NSE pods.
  2. Keep the setup untouched for a while and possibly run the traffic between client and endpoint pods.
  3. You get the idea...

Context

Failure Logs

Forwarder logs:

Aug 16 08:29:57.503 [INFO] [name:forwarder-ovs-bvjm6] (1.1)   request={"connection":{"id":"b2dd2988-2b73-457e-80eb-82724a67505f","network_service":"icmp-responder","mechanism":{"cls":"REMOTE","type":"VXLAN","parameters":{"dst_ip":"10.10.10.33","dst_port":"4789","src_ip":"10.10.10.22","src_port":"4789","vni":"9070652"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-cb878b7cf-xg2sb","id":"nsc-kernel-cb878b7cf-xg2sb-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMwMzEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.m1KXC1e4COyI0wBO0oXcG8LoX509f7KoP-5FGGfmloZJ15yGq6NAB2keEPQnCRm36V1N4pHn4AZ4OtScCvwlrg","expires":{"seconds":1629103031,"nanos":904438128}},{"name":"nsmgr-52tmw","id":"97560283-dd11-4eda-b1c1-0627ff771784","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMwMzIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.4xq-CNZ1zXG726xOH8BQ-KtTWGxQKq_ydcmafVT9OzCEgGhIZSyMkiYkd5vedgsip6WcQBCiLp-s024ZMX1XBA","expires":{"seconds":1629103032,"nanos":4033021}},{"name":"forwarder-ovs-grmg5","id":"bc44d0f4-0fd6-48e7-951c-a0f17adeb113","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDQ5MTAsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.LmEigJIPaPFEzkHUEwpVwokY4QeC0IhBSJYsptWQLrSRN9h2u_e8anQkA5oQIptgnh_9Wrd-r3kXvS0cyQKA_Q","expires":{"seconds":1629104910}},{"name":"nsmgr-52tmw","id":"c063a9eb-788b-4dec-9b82-63e45fa66aab","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.eZAQacvJML1NCpEpvoIk4IgVKcpPUQOV-DiIxjq5Qm_j34K8z-9adD_NnyOyqYgz34k2HxoCFG6ZyZe4XseO7g","expires":{"seconds":1629103197,"nanos":398897352}},{"name":"nsmgr-7xdp2","id":"b2dd2988-2b73-457e-80eb-82724a67505f","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.Jo1DIzMvDdDu4ut-9H6_3kFgdUQDxcuZIYcR7M-5_xSnJEsQXEujRuw9t0nR9huOwBEeEqZXhVDyrsk4CY5f4g","expires":{"seconds":1629103197,"nanos":402083304}},{"name":"forwarder-ovs-bvjm6","id":"68c8c60b-aae3-4c7f-a736-ce4a35db732c","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDQ5OTIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.BNQ2Mic83a98rEs8HX_AqSbW8emgZGqUYNEPxdgRhIPj9xlflE1Pf2Cnr7DPugUJr97s0NgE1BhFOFOi2LpAyA","expires":{"seconds":1629104992}},{"name":"nsmgr-7xdp2","id":"13f9d5fa-c5b8-4259-af22-45fdd02bc580","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MTAzMDMyLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.W3dcWtrmws-OSAPZ_5UQ-iqtS-Q_frDxezWuheXgkaSFNzvzlo6MiFt5Cw0kkyN-Xv8viyKzkfjSqsQWVJyrkw","expires":{"seconds":1629103032,"nanos":304457798}},{"name":"nse-kernel-945c556db-vbblb","id":"b33cdb74-06dd-4ce0-8669-00227aea9209","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMwMzIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.8HOZ20O3yN13zVF0RDpzYxiB_znBpJ8y2NFXkQWglxegTqmen25b8RXUo5mNDXv_EVHNlUR5-FZ6877ks4sruA","expires":{"seconds":1629103032,"nanos":307027675}}]},"network_service_endpoint_name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
Aug 16 08:29:57.503 [INFO] [name:forwarder-ovs-bvjm6] (1.2)   request-diff={"connection":{"id":"68c8c60b-aae3-4c7f-a736-ce4a35db732c","path":{"index":5,"path_segments":{"4":{"expires":{"nanos":502907557},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.f-1MkV7Iak4pEtUUQNOBxsLjtVidl7hCpPDnHgLq1ZbuF7MIRFk1J00IKXV7eFbMpc3PA01cp_gWp63aWFTNBQ"}}}}} span=406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (2)  ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=406cdaa5e60e81a2:52338a96b3ce83e9:56edd9f2d5712279:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (3)   ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=406cdaa5e60e81a2:1c10e9ede3bf742a:52338a96b3ce83e9:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (4)    ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=406cdaa5e60e81a2:266fcaf4a3af11f2:1c10e9ede3bf742a:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (5)     ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=406cdaa5e60e81a2:79fe37457dce6fa3:266fcaf4a3af11f2:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (6)      ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=406cdaa5e60e81a2:60118e58fcd9b69e:79fe37457dce6fa3:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (7)       ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (7.1)         request-diff={"connection":{"path":{"path_segments":{"5":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDQ5OTIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.dDNFKVHghBhHuphf7KROTaiCIcQncbCF-AFWbpcJn5SLWJYUj5axmwgUaK5S3gyiBPFaYGPioRR5NtkS0Osq2g"}}}}} span=406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (8)        ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=406cdaa5e60e81a2:20e005e6da164b4e:47292580b1921315:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (9)         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=406cdaa5e60e81a2:191949718b4ad4c5:20e005e6da164b4e:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=406cdaa5e60e81a2:5333ce0813706551:191949718b4ad4c5:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (11)           ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=406cdaa5e60e81a2:0d1c3b85f6bcca6f:5333ce0813706551:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (12)            ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=406cdaa5e60e81a2:28ce5cdd432321ee:0d1c3b85f6bcca6f:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=406cdaa5e60e81a2:27aeea0d0c771228:28ce5cdd432321ee:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (14)              ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=406cdaa5e60e81a2:67a200715d6291bc:27aeea0d0c771228:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (15)               ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=406cdaa5e60e81a2:66dbcb9e112799fe:67a200715d6291bc:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (16)                ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=406cdaa5e60e81a2:05242dd3140a274b:66dbcb9e112799fe:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (17)                 ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=406cdaa5e60e81a2:449d9837ab879a96:05242dd3140a274b:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (18)                  ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=406cdaa5e60e81a2:1bac7b46eabeb2d2:449d9837ab879a96:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (19)                   ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=406cdaa5e60e81a2:5161149fda649b5b:1bac7b46eabeb2d2:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (20)                    ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=406cdaa5e60e81a2:24ccaa524beaae0a:5161149fda649b5b:1
Aug 16 08:29:57.508 [INFO] [name:forwarder-ovs-bvjm6] (21)                     ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=406cdaa5e60e81a2:7abea4adbbac6e28:24ccaa524beaae0a:1
Aug 16 08:29:57.508 [INFO] [name:forwarder-ovs-bvjm6] (22)                      ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
Aug 16 08:29:57.508 [INFO] [name:forwarder-ovs-bvjm6] (23)                       ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=406cdaa5e60e81a2:321e4097129c2390:32b6d8b3862ab7df:1
Aug 16 08:29:57.509 [ERRO] [name:forwarder-ovs-bvjm6] (23.1)                         context canceled
Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:60
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress.(*ipaddressServer).Request
        /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/server.go:64
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes.(*routesServer).Request
        /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors.(*ipNeighborsServer).Request
        /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/server.go:37
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu.(*mtuServer).Request
        /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69 span=406cdaa5e60e81a2:321e4097129c2390:32b6d8b3862ab7df:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:321e4097129c2390:32b6d8b3862ab7df:1
Aug 16 08:29:57.509 [INFO] [name:forwarder-ovs-bvjm6] (22.1)                        response=null span=406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
Aug 16 08:29:57.510 [ERRO] [name:forwarder-ovs-bvjm6] (22.2)                        Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
Aug 16 08:29:57.511 [ERRO] [name:forwarder-ovs-bvjm6] (21.1)                       Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:7abea4adbbac6e28:24ccaa524beaae0a:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:7abea4adbbac6e28:24ccaa524beaae0a:1
Aug 16 08:29:57.512 [ERRO] [name:forwarder-ovs-bvjm6] (20.1)                      Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:24ccaa524beaae0a:5161149fda649b5b:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:24ccaa524beaae0a:5161149fda649b5b:1
Aug 16 08:29:57.513 [ERRO] [name:forwarder-ovs-bvjm6] (19.1)                     Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5161149fda649b5b:1bac7b46eabeb2d2:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5161149fda649b5b:1bac7b46eabeb2d2:1
Aug 16 08:29:57.514 [ERRO] [name:forwarder-ovs-bvjm6] (18.1)                    Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:1bac7b46eabeb2d2:449d9837ab879a96:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:1bac7b46eabeb2d2:449d9837ab879a96:1
Aug 16 08:29:57.515 [ERRO] [name:forwarder-ovs-bvjm6] (17.1)                   Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:449d9837ab879a96:05242dd3140a274b:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:449d9837ab879a96:05242dd3140a274b:1
Aug 16 08:29:57.516 [ERRO] [name:forwarder-ovs-bvjm6] (16.1)                  Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:05242dd3140a274b:66dbcb9e112799fe:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:05242dd3140a274b:66dbcb9e112799fe:1
Aug 16 08:29:57.516 [ERRO] [name:forwarder-ovs-bvjm6] (15.1)                 Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:66dbcb9e112799fe:67a200715d6291bc:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:66dbcb9e112799fe:67a200715d6291bc:1
Aug 16 08:29:57.517 [ERRO] [name:forwarder-ovs-bvjm6] (14.1)                Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:67a200715d6291bc:27aeea0d0c771228:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:67a200715d6291bc:27aeea0d0c771228:1
Aug 16 08:29:57.518 [ERRO] [name:forwarder-ovs-bvjm6] (13.1)               Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:27aeea0d0c771228:28ce5cdd432321ee:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:27aeea0d0c771228:28ce5cdd432321ee:1
Aug 16 08:29:57.518 [ERRO] [name:forwarder-ovs-bvjm6] (12.1)              Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:28ce5cdd432321ee:0d1c3b85f6bcca6f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:28ce5cdd432321ee:0d1c3b85f6bcca6f:1
Aug 16 08:29:57.519 [ERRO] [name:forwarder-ovs-bvjm6] (11.2)             Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:0d1c3b85f6bcca6f:5333ce0813706551:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:0d1c3b85f6bcca6f:5333ce0813706551:1
Aug 16 08:29:57.521 [ERRO] [name:forwarder-ovs-bvjm6] (10.1)            Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5333ce0813706551:191949718b4ad4c5:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5333ce0813706551:191949718b4ad4c5:1
Aug 16 08:29:57.521 [ERRO] [name:forwarder-ovs-bvjm6] (9.1)           Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:191949718b4ad4c5:20e005e6da164b4e:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:191949718b4ad4c5:20e005e6da164b4e:1
Aug 16 08:29:57.522 [ERRO] [name:forwarder-ovs-bvjm6] (8.1)          Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:20e005e6da164b4e:47292580b1921315:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:20e005e6da164b4e:47292580b1921315:1
Aug 16 08:29:57.522 [ERRO] [name:forwarder-ovs-bvjm6] (7.2)         Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
Aug 16 08:29:57.522 [ERRO] [name:forwarder-ovs-bvjm6] (6.1)        Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:60118e58fcd9b69e:79fe37457dce6fa3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:60118e58fcd9b69e:79fe37457dce6fa3:1
Aug 16 08:29:57.523 [ERRO] [name:forwarder-ovs-bvjm6] (5.1)       Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:79fe37457dce6fa3:266fcaf4a3af11f2:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:79fe37457dce6fa3:266fcaf4a3af11f2:1
Aug 16 08:29:57.523 [ERRO] [name:forwarder-ovs-bvjm6] (4.1)      Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:266fcaf4a3af11f2:1c10e9ede3bf742a:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:266fcaf4a3af11f2:1c10e9ede3bf742a:1
Aug 16 08:29:57.523 [ERRO] [name:forwarder-ovs-bvjm6] (3.1)     Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:1c10e9ede3bf742a:52338a96b3ce83e9:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:1c10e9ede3bf742a:52338a96b3ce83e9:1
Aug 16 08:29:57.524 [ERRO] [name:forwarder-ovs-bvjm6] (2.1)    Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:52338a96b3ce83e9:56edd9f2d5712279:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:52338a96b3ce83e9:56edd9f2d5712279:1
Aug 16 08:29:57.524 [ERRO] [name:forwarder-ovs-bvjm6] (1.3)   Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:6dbba770b04a0379:77df86fd1c366362:1

NSMgr logs:

Aug 16 08:29:57.400 [INFO] [name:nsmgr-7xdp2] (1.2)   request-diff={"connection":{"id":"b2dd2988-2b73-457e-80eb-82724a67505f","path":{"index":4,"path_segments":{"3":{"expires":{"nanos":398897352,"seconds":1629103197},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.eZAQacvJML1NCpEpvoIk4IgVKcpPUQOV-DiIxjq5Qm_j34K8z-9adD_NnyOyqYgz34k2HxoCFG6ZyZe4XseO7g"}}}}} span=406cdaa5e60e81a2:00a4f28db9521d81:70606533bfc55800:1
Aug 16 08:29:57.400 [INFO] [name:nsmgr-7xdp2] (2)  ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=406cdaa5e60e81a2:016f320a847004f0:00a4f28db9521d81:1
Aug 16 08:29:57.400 [INFO] [name:nsmgr-7xdp2] (3)   ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=406cdaa5e60e81a2:3e2fa8b80cb704d3:016f320a847004f0:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (4)    ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=406cdaa5e60e81a2:3158c52cd1f82f1c:3e2fa8b80cb704d3:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (5)     ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=406cdaa5e60e81a2:5be08f649388c434:3158c52cd1f82f1c:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (6)      ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=406cdaa5e60e81a2:13f49d421954e6c0:5be08f649388c434:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (7)       ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
Aug 16 08:29:57.402 [INFO] [name:nsmgr-7xdp2] (7.1)         request-diff={"connection":{"path":{"path_segments":{"4":{"expires":{"nanos":402083304,"seconds":1629103197},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.Jo1DIzMvDdDu4ut-9H6_3kFgdUQDxcuZIYcR7M-5_xSnJEsQXEujRuw9t0nR9huOwBEeEqZXhVDyrsk4CY5f4g"}}}}} span=406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (8)        ⎆ sdk/pkg/networkservice/core/adapters/clientToServer.Request() span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [WARN] [name:nsmgr-7xdp2] (8.1)          Environment variable CLUSTER_NAME is not set. Skipping. span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [WARN] [name:nsmgr-7xdp2] (8.2)          The label nodeName was already assigned to dl380-004. Skipping. span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [WARN] [name:nsmgr-7xdp2] (8.3)          Environment variable POD_NAME is not set. Skipping. span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (9)         ⎆ sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request() span=406cdaa5e60e81a2:08c02d56e082bd1d:23793339b1f8e6c7:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (10)          ⎆ sdk/pkg/registry/common/clienturl/clientURLNSEServer.Find() span=406cdaa5e60e81a2:5b54b23309b8047f:08c02d56e082bd1d:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (10.1)            find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"}} span=406cdaa5e60e81a2:5b54b23309b8047f:08c02d56e082bd1d:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (11)           ⎆ sdk/pkg/registry/common/connect/connectNSEServer.Find() span=406cdaa5e60e81a2:1165320eb7e92320:5b54b23309b8047f:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (11.1)             find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"}} span=406cdaa5e60e81a2:1165320eb7e92320:5b54b23309b8047f:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (12)            ⎆ sdk/pkg/registry/common/connect/grpcNSEClient.Find() span=406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (12.1)              find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"}} span=406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
Aug 16 08:29:57.404 [INFO] [name:nsmgr-7xdp2] (12.2)              response={"ClientStream":{"ClientStream":{}}} span=406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
Aug 16 08:29:57.404 [INFO] [name:nsmgr-7xdp2] (13)             ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=406cdaa5e60e81a2:336a9d5e80f2e612:0b2a7e94aaec15b8:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (13.1)               response={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=406cdaa5e60e81a2:336a9d5e80f2e612:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:336a9d5e80f2e612:0b2a7e94aaec15b8:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (12)            ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=406cdaa5e60e81a2:59732a0aa84259f3:1165320eb7e92320:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (12.1)              network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=406cdaa5e60e81a2:59732a0aa84259f3:1165320eb7e92320:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (11)           ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=406cdaa5e60e81a2:6978090a448e08ca:5b54b23309b8047f:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (11.1)             network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=406cdaa5e60e81a2:6978090a448e08ca:5b54b23309b8047f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:6978090a448e08ca:5b54b23309b8047f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:59732a0aa84259f3:1165320eb7e92320:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (13)             ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=406cdaa5e60e81a2:4eb03cbc01f5ad16:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4abe12adcda1a071:0b2a7e94aaec15b8:1
Aug 16 08:29:57.410 [ERRO] [name:nsmgr-7xdp2] (13.1)               EOF
Error returned from api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryFindClient).Recv
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/trace/nse_registry.go:52
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.nseFindClientToServer
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_registry.go:107
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*networkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_registry.go:52
github.com/networkservicemesh/sdk/pkg/registry/common/connect.(*connectNSEServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/common/connect/nse_server.go:98
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/trace/nse_registry.go:174
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_server.go:71
github.com/networkservicemesh/sdk/pkg/registry/common/clienturl.(*clientURLNSEServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/common/clienturl/nse_server.go:49
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/trace/nse_registry.go:174
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_server.go:62
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_server.go:62
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.nseFindServerToClient
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_registry.go:132
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*networkServiceEndpointRegistryClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_registry.go:85
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_client.go:49
github.com/networkservicemesh/sdk/pkg/registry/common/querycache.(*queryCacheNSEClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/common/querycache/nse_client.go:58
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_client.go:47
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*callNextNSEClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_next_client.go:35
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_server.go:71
github.com/networkservicemesh/sdk/pkg/registry/common/localbypass.(*localBypassNSEServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/common/localbypass/server.go:77
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_server.go:62
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.nseFindServerToClient
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_registry.go:132
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*networkServiceEndpointRegistryClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/adapters/nse_registry.go:85
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_client.go:47
github.com/networkservicemesh/sdk/pkg/registry/common/serialize.(*serializeNSEClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/common/serialize/nse_client.go:50
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/registry/core/next/nse_registry_client.go:47
github.com/networkservicemesh/sdk/pkg/networkservice/common/discover.(*discoverCandidatesServer).discoverNetworkServiceEndpoint
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/common/discover/server.go:145
github.com/networkservicemesh/sdk/pkg/networkservice/common/discover.(*discoverCandidatesServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/common/discover/server.go:61
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/adapters.(*callNextServer).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/adapters/client_to_server.go:53 span=406cdaa5e60e81a2:4eb03cbc01f5ad16:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4eb03cbc01f5ad16:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:1165320eb7e92320:5b54b23309b8047f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5b54b23309b8047f:08c02d56e082bd1d:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (2)  ⎆ sdk/pkg/registry/common/clienturl/clientURLNSEServer.Find() span=37c32fda70cfd60a:774519f395ee7ba6:37c32fda70cfd60a:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (2.1)    find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"},"watch":true} span=37c32fda70cfd60a:774519f395ee7ba6:37c32fda70cfd60a:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (3)   ⎆ sdk/pkg/registry/common/connect/connectNSEServer.Find() span=37c32fda70cfd60a:477fa58f6514c26b:774519f395ee7ba6:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (3.1)     find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"},"watch":true} span=37c32fda70cfd60a:477fa58f6514c26b:774519f395ee7ba6:1
Aug 16 08:29:57.410 [INFO] [name:nsmgr-7xdp2] (10)          ⎆ sdk/pkg/networkservice/common/roundrobin/selectEndpointServer.Request() span=406cdaa5e60e81a2:3a302fe2e7c85311:08c02d56e082bd1d:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (4)    ⎆ sdk/pkg/registry/common/connect/grpcNSEClient.Find() span=37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (4.1)      find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"},"watch":true} span=37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (4.2)      response={"ClientStream":{"ClientStream":{}}} span=37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
Aug 16 08:29:57.411 [INFO] [cmd:Nsmgr] (5)     ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=37c32fda70cfd60a:072eb83ff79856ba:0b9b73968be5bc1c:1
Aug 16 08:29:57.411 [INFO] [name:nsmgr-7xdp2] (11)           ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesServer.Request() span=406cdaa5e60e81a2:4a597879c775d5e9:3a302fe2e7c85311:1
Aug 16 08:29:57.411 [INFO] [name:nsmgr-7xdp2] (12)            ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=406cdaa5e60e81a2:033390a7d018b022:4a597879c775d5e9:1
Aug 16 08:29:57.411 [INFO] [name:nsmgr-7xdp2] (13)             ⎆ sdk/pkg/networkservice/common/interpose/interposeServer.Request() span=406cdaa5e60e81a2:3460afd7c9dc355f:033390a7d018b022:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (5.1)       response={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:072eb83ff79856ba:0b9b73968be5bc1c:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:072eb83ff79856ba:0b9b73968be5bc1c:1
Aug 16 08:29:57.500 [INFO] [name:nsmgr-7xdp2] (14)              ⎆ sdk/pkg/networkservice/common/filtermechanisms/filterMechanismsServer.Request() span=406cdaa5e60e81a2:5ea8f8ccdea881e0:3460afd7c9dc355f:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4)    ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:638a2882e63ead96:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4.1)      network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:638a2882e63ead96:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (3)   ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:7f9fa62075e341fe:774519f395ee7ba6:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (3.1)     network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:7f9fa62075e341fe:774519f395ee7ba6:1
Aug 16 08:29:57.500 [INFO] [name:nsmgr-7xdp2] (15)               ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=406cdaa5e60e81a2:54d359380244f3b3:5ea8f8ccdea881e0:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:7f9fa62075e341fe:774519f395ee7ba6:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:638a2882e63ead96:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (5)     ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=37c32fda70cfd60a:0d13a554f914f1c0:0b9b73968be5bc1c:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (5.1)       response={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:0d13a554f914f1c0:0b9b73968be5bc1c:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:0d13a554f914f1c0:0b9b73968be5bc1c:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4)    ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:41644ae95fa9f421:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4.1)      network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:41644ae95fa9f421:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (3)   ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:713c0a89d0077871:774519f395ee7ba6:1
Aug 16 08:29:57.501 [INFO] [cmd:Nsmgr] (3.1)     network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:713c0a89d0077871:774519f395ee7ba6:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:713c0a89d0077871:774519f395ee7ba6:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:41644ae95fa9f421:477fa58f6514c26b:1
Aug 16 08:29:57.501 [INFO] [cmd:Nsmgr] (5)     ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=37c32fda70cfd60a:15a165de952b654c:0b9b73968be5bc1c:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (16)                ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=406cdaa5e60e81a2:38a7ac8a3d90e738:54d359380244f3b3:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (17)                 ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request() span=406cdaa5e60e81a2:12d66789fd896f67:38a7ac8a3d90e738:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (18)                  ⎆ sdk/pkg/networkservice/common/serialize/serializeClient.Request() span=406cdaa5e60e81a2:4692d25faa6611cf:12d66789fd896f67:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (19)                   ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request() span=406cdaa5e60e81a2:548bf9a8352571ff:4692d25faa6611cf:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (20)                    ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request() span=406cdaa5e60e81a2:5e5acdd1a7dbc069:548bf9a8352571ff:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (21)                     ⎆ sdk/pkg/networkservice/common/heal/healClient.Request() span=406cdaa5e60e81a2:2f40b5a0da0f00a3:5e5acdd1a7dbc069:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (22)                      ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request() span=406cdaa5e60e81a2:50d728211e48ca9f:2f40b5a0da0f00a3:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (23)                       ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request() span=406cdaa5e60e81a2:0ef2bb586c60e040:50d728211e48ca9f:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (24)                        ⎆ sdk/pkg/networkservice/common/null/nullClient.Request() span=406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (25)                         ⎆ api/pkg/api/networkservice/networkServiceClient.Request() span=406cdaa5e60e81a2:3e7fb8bed835e4e9:32bd5e99745b57f7:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:77df86fd1c366362:3e7fb8bed835e4e9:1
Aug 16 08:29:57.525 [ERRO] [name:nsmgr-7xdp2] (25.1)                           rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled
Error returned from api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:62
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/null.(*nullClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/common/null/client.go:43
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/common/mechanisms/sendfd/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.(*recvFDClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/common/mechanisms/recvfd/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/heal.(*healClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/common/heal/client.go:106
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/utils/metadata.(*metaDataClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/utils/metadata/client.go:45
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
        /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210802230059-fe8d8e6e825a/pkg/networkservice/core/next/client.go:69 span=406cdaa5e60e81a2:3e7fb8bed835e4e9:32bd5e99745b57f7:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3e7fb8bed835e4e9:32bd5e99745b57f7:1
Aug 16 08:29:57.525 [INFO] [name:nsmgr-7xdp2] (24.1)                          response=null span=406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
Aug 16 08:29:57.525 [ERRO] [name:nsmgr-7xdp2] (24.2)                          Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
Aug 16 08:29:57.525 [ERRO] [name:nsmgr-7xdp2] (23.1)                         Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:0ef2bb586c60e040:50d728211e48ca9f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:0ef2bb586c60e040:50d728211e48ca9f:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (22.1)                        Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:50d728211e48ca9f:2f40b5a0da0f00a3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:50d728211e48ca9f:2f40b5a0da0f00a3:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (21.1)                       Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:2f40b5a0da0f00a3:5e5acdd1a7dbc069:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:2f40b5a0da0f00a3:5e5acdd1a7dbc069:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (20.1)                      Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5e5acdd1a7dbc069:548bf9a8352571ff:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5e5acdd1a7dbc069:548bf9a8352571ff:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (19.1)                     Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:548bf9a8352571ff:4692d25faa6611cf:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:548bf9a8352571ff:4692d25faa6611cf:1
Aug 16 08:29:57.527 [ERRO] [name:nsmgr-7xdp2] (18.1)                    Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:4692d25faa6611cf:12d66789fd896f67:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4692d25faa6611cf:12d66789fd896f67:1
2021/08/16 08:29:57 ERROR: error reporting Jaeger span "sdk/pkg/networkservice/common/serialize/serializeClient.Request": write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
Aug 16 08:29:57.600 [ERRO] [name:nsmgr-7xdp2] (17.1)                   Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:12d66789fd896f67:38a7ac8a3d90e738:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:12d66789fd896f67:38a7ac8a3d90e738:1
Aug 16 08:29:57.601 [ERRO] [name:nsmgr-7xdp2] (16.1)                  Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:38a7ac8a3d90e738:54d359380244f3b3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:38a7ac8a3d90e738:54d359380244f3b3:1
Aug 16 08:29:57.602 [ERRO] [name:nsmgr-7xdp2] (15.1)                 Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:54d359380244f3b3:5ea8f8ccdea881e0:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:54d359380244f3b3:5ea8f8ccdea881e0:1
Aug 16 08:29:57.603 [ERRO] [name:nsmgr-7xdp2] (14.1)                Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5ea8f8ccdea881e0:3460afd7c9dc355f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5ea8f8ccdea881e0:3460afd7c9dc355f:1
2021/08/16 08:29:57 ERROR: error reporting Jaeger span "sdk/pkg/networkservice/common/filtermechanisms/filterMechanismsServer.Request": write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
Aug 16 08:29:57.604 [ERRO] [name:nsmgr-7xdp2] (13.2)               Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3460afd7c9dc355f:033390a7d018b022:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3460afd7c9dc355f:033390a7d018b022:1
Aug 16 08:29:57.604 [ERRO] [name:nsmgr-7xdp2] (12.1)              Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:033390a7d018b022:4a597879c775d5e9:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:033390a7d018b022:4a597879c775d5e9:1
Aug 16 08:29:57.605 [ERRO] [name:nsmgr-7xdp2] (11.2)             Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:4a597879c775d5e9:3a302fe2e7c85311:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4a597879c775d5e9:3a302fe2e7c85311:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (10.1)            Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3a302fe2e7c85311:08c02d56e082bd1d:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3a302fe2e7c85311:08c02d56e082bd1d:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (9.2)           Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:08c02d56e082bd1d:23793339b1f8e6c7:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:08c02d56e082bd1d:23793339b1f8e6c7:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (8.4)          Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (7.2)         Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (6.1)        Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:13f49d421954e6c0:5be08f649388c434:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:13f49d421954e6c0:5be08f649388c434:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (5.1)       Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5be08f649388c434:3158c52cd1f82f1c:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5be08f649388c434:3158c52cd1f82f1c:1
2021/08/16 08:29:57 ERROR: error reporting Jaeger span "sdk/pkg/networkservice/common/monitor/monitorServer.Request": write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (4.1)      Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3158c52cd1f82f1c:3e2fa8b80cb704d3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3158c52cd1f82f1c:3e2fa8b80cb704d3:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (3.1)     Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3e2fa8b80cb704d3:016f320a847004f0:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3e2fa8b80cb704d3:016f320a847004f0:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (2.1)    Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:016f320a847004f0:00a4f28db9521d81:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:016f320a847004f0:00a4f28db9521d81:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (1.3)   Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:00a4f28db9521d81:70606533bfc55800:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:00a4f28db9521d81:70606533bfc55800:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:70606533bfc55800:41caa70e1b77d793:1
2021/08/16 08:29:59 Reporting span 36ba8cd186c2dc89:36ba8cd186c2dc89:0000000000000000:1
2021/08/16 08:29:59 ERROR: failed to flush Jaeger spans to server: write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
2021/08/16 08:30:06 Reporting span 7d10e663f026ed3f:7d10e663f026ed3f:0000000000000000:1
pperiyasamy commented 3 years ago

@Bolodya1997 as discussed, i tried with latest NSM images and issue still exists.

Here is the forwarder log:

Aug 17 12:54:42.219 [INFO] [name:forwarder-ovs-jck5d] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
Aug 17 12:54:42.219 [INFO] [name:forwarder-ovs-jck5d] (1.1)   request={"connection":{"id":"a5052de1-cb38-44c0-ae11-edb4730ae064","network_service":"icmp-responder","mechanism":{"cls":"REMOTE","type":"VXLAN","parameters":{"dst_ip":"10.10.10.33","dst_port":"4789","src_ip":"10.10.10.22","src_port":"4789","vni":"9832580"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.0uXakzic0NC44YVPyyDA_32jy9qQrIShPT5E5Hva0KWZOSBts0FthOtLu7JcuiHrQ_XQr30oF7ByUrpvm4s48Q","expires":{"seconds":1629205482,"nanos":103258984}},{"name":"forwarder-ovs-vc7qb","id":"df936088-582e-41db-8205-32e8d8b849b6","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.YRK2sbi52Tk7OfIGhocjZrHq_nOAXlgCg9-tceigiyK5bsAHMAe_1oMBmZswke2eNVpMn1bcF7mdHUzMH3PuHg","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"5be19c57-bff5-467d-be43-6eb2853c83e2","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.UlrMLPTy01M7Bjbded8Fr1O-hRtwa1-bweunvEfZInYy_fGxBx37Osrwvw42Zckz4lSpkS-JLuKzdJTdWBm7tg","expires":{"seconds":1629205482,"nanos":205443660}},{"name":"nsmgr-8hs7f","id":"a5052de1-cb38-44c0-ae11-edb4730ae064","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.d2D6DxI7rJZhU0AQsb-jiohPN8mBbxeFmofCdJ5hiEF4uTW5uEzOJzO30MmHgMwL8nmVmU3k6XJO6Agji4d3GA","expires":{"seconds":1629205482,"nanos":208094292}},{"name":"forwarder-ovs-jck5d","id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.3R2_rMVDKSdrYY5Isrh48BK2zVelfNuf8Zu7GCNJOmCZVJFmfSPYFBW6hWKqvemaTwcHqvIeIZTvraOIV1_vKQ","expires":{"seconds":1629207419}},{"name":"nsmgr-8hs7f","id":"fbcd7003-f6ad-471b-9bad-f212e54566cd","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MjA1MzYxLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.CH4-Zhfd-pWO14H7pVgoBXsvBbXLrKlwHUR0Zd9--LCcUMO-OsxqJtOlzg9VlkbVeoph0Ad8duTd1MkqYNFX1Q","expires":{"seconds":1629205361,"nanos":420051348}},{"name":"nse-kernel-6444cdcc55-jjjfv","id":"356b02b4-7bc0-45fe-86e2-70ccd4358632","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDUzNjEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.duPMH9KJYAS75akCaUIc0wHssP73AcBQgq_B6P96_lcyoKnmi2ocXQ64X0DrqnhhbjZqS9uOKgFbDIQFiJk92g","expires":{"seconds":1629205361,"nanos":422670822}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
Aug 17 12:54:42.219 [INFO] [name:forwarder-ovs-jck5d] (1.2)   request-diff={"connection":{"id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","path":{"index":5,"path_segments":{"4":{"expires":{"nanos":217344406},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.il2Xax2TBZ6CATGWNuQvF4D5O-nlv7LYTkfhw8tzl-lShp3OM-J5qVk0mmzJ5_Xfkko-lNrGakfI3nf0GTYrvQ"}}}}} span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
Aug 17 12:54:42.219 [INFO] [name:forwarder-ovs-jck5d] (2)  ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=03e9f52b95487f7f:3d9c60d29dff250f:7a7cce123fb62752:1
Aug 17 12:54:42.220 [INFO] [name:forwarder-ovs-jck5d] (3)   ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=03e9f52b95487f7f:609c63bb3c8de1a7:3d9c60d29dff250f:1
Aug 17 12:54:42.221 [INFO] [name:forwarder-ovs-jck5d] (4)    ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=03e9f52b95487f7f:76f00b136766d430:609c63bb3c8de1a7:1
Aug 17 12:54:42.221 [INFO] [name:forwarder-ovs-jck5d] (5)     ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=03e9f52b95487f7f:6ee58b63ac81944e:76f00b136766d430:1
Aug 17 12:54:42.222 [INFO] [name:forwarder-ovs-jck5d] (6)      ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=03e9f52b95487f7f:7190c8ee82847f23:6ee58b63ac81944e:1
Aug 17 12:54:42.222 [INFO] [name:forwarder-ovs-jck5d] (7)       ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=03e9f52b95487f7f:485fb6841c630633:7190c8ee82847f23:1
Aug 17 12:54:42.222 [INFO] [name:forwarder-ovs-jck5d] (7.1)         request-diff={"connection":{"path":{"path_segments":{"5":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.oqz1j0DMd4uHsb2LyuRoCFJe9pcskLy5lZ-H5_6Jpc-xNBBhFeumRVba_R57FzpCWD3W-bn0PwrKBHR__AoWAw"}}}}} span=03e9f52b95487f7f:485fb6841c630633:7190c8ee82847f23:1
Aug 17 12:54:42.222 [INFO] [name:forwarder-ovs-jck5d] (8)        ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=03e9f52b95487f7f:209c9732b7709b77:485fb6841c630633:1
Aug 17 12:54:42.222 [INFO] [name:forwarder-ovs-jck5d] (9)         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=03e9f52b95487f7f:303a460c70ffe464:209c9732b7709b77:1
Aug 17 12:54:42.223 [INFO] [name:forwarder-ovs-jck5d] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=03e9f52b95487f7f:04996bd5df684e5c:303a460c70ffe464:1
Aug 17 12:54:42.223 [INFO] [name:forwarder-ovs-jck5d] (11)           ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=03e9f52b95487f7f:12753f2931c02cb3:04996bd5df684e5c:1
Aug 17 12:54:42.223 [INFO] [name:forwarder-ovs-jck5d] (12)            ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=03e9f52b95487f7f:5fff97c6694d58a1:12753f2931c02cb3:1
Aug 17 12:54:42.223 [INFO] [name:forwarder-ovs-jck5d] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=03e9f52b95487f7f:25468e3012761c18:5fff97c6694d58a1:1
Aug 17 12:54:42.223 [INFO] [name:forwarder-ovs-jck5d] (14)              ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=03e9f52b95487f7f:2a271d7f7444fb04:25468e3012761c18:1
Aug 17 12:54:42.223 [INFO] [name:forwarder-ovs-jck5d] (15)               ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=03e9f52b95487f7f:6365ab165c9d2342:2a271d7f7444fb04:1
Aug 17 12:54:42.224 [INFO] [name:forwarder-ovs-jck5d] (16)                ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=03e9f52b95487f7f:720a4aff192ac769:6365ab165c9d2342:1
Aug 17 12:54:42.224 [INFO] [name:forwarder-ovs-jck5d] (17)                 ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=03e9f52b95487f7f:63d6a7283737ab3c:720a4aff192ac769:1
Aug 17 12:54:42.224 [INFO] [name:forwarder-ovs-jck5d] (18)                  ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=03e9f52b95487f7f:4cfad252feee66ca:63d6a7283737ab3c:1
Aug 17 12:54:42.225 [INFO] [name:forwarder-ovs-jck5d] (19)                   ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=03e9f52b95487f7f:09678d30a096de49:4cfad252feee66ca:1
Aug 17 12:54:42.225 [INFO] [name:forwarder-ovs-jck5d] (20)                    ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=03e9f52b95487f7f:584d665bbbbc6573:09678d30a096de49:1
Aug 17 12:54:42.225 [INFO] [name:forwarder-ovs-jck5d] (21)                     ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=03e9f52b95487f7f:08776bc38c65c580:584d665bbbbc6573:1
Aug 17 12:54:42.225 [INFO] [name:forwarder-ovs-jck5d] (22)                      ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=03e9f52b95487f7f:0e18efecba32b8ea:08776bc38c65c580:1
Aug 17 12:54:42.226 [INFO] [name:forwarder-ovs-jck5d] (23)                       ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=03e9f52b95487f7f:527ba635996fad7d:0e18efecba32b8ea:1
Aug 17 12:54:42.226 [INFO] [name:forwarder-ovs-jck5d] (24)                        ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request() span=03e9f52b95487f7f:7e05a0d199774e24:527ba635996fad7d:1
Aug 17 12:54:42.226 [INFO] [name:forwarder-ovs-jck5d] (25)                         ⎆ sdk/pkg/networkservice/common/serialize/serializeClient.Request() span=03e9f52b95487f7f:541f71b4a17c3c8f:7e05a0d199774e24:1
Aug 17 12:54:42.227 [INFO] [name:forwarder-ovs-jck5d] (26)                          ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request() span=03e9f52b95487f7f:19fb894ed9f7f635:541f71b4a17c3c8f:1
Aug 17 12:54:42.227 [INFO] [name:forwarder-ovs-jck5d] (27)                           ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request() span=03e9f52b95487f7f:3b56478b9a5c05c8:19fb894ed9f7f635:1
Aug 17 12:54:42.227 [INFO] [name:forwarder-ovs-jck5d] (28)                            ⎆ sdk/pkg/networkservice/common/heal/healClient.Request() span=03e9f52b95487f7f:2eee0b6eb144c851:3b56478b9a5c05c8:1
Aug 17 12:54:42.228 [INFO] [name:forwarder-ovs-jck5d] (29)                             ⎆ sdk/pkg/networkservice/common/mechanismtranslation/mechanismTranslationClient.Request() span=03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
Aug 17 12:54:42.228 [INFO] [name:forwarder-ovs-jck5d] (29.1)                               request-diff={"connection":{"mechanism":{"cls":"LOCAL","parameters":{"+inodeURL":"file:///proc/51245/fd/12","+name":"icmp-respo-356b","-dst_ip":"10.10.10.33","-dst_port":"4789","-src_ip":"10.10.10.22","-src_port":"4789","-vni":"9832580"},"type":"KERNEL"}},"mechanism_preferences":{"-0":{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}}} span=03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
Aug 17 12:54:42.228 [INFO] [name:forwarder-ovs-jck5d] (30)                              ⎆ sdk-ovs/pkg/networkservice/l2ovsconnect/l2ConnectClient.Request() span=03e9f52b95487f7f:7cd300ebba7d3cee:3fbd6ab211997370:1
Aug 17 12:54:42.228 [INFO] [name:forwarder-ovs-jck5d] (31)                               ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=03e9f52b95487f7f:0ff81ec2ba0aa6bf:7cd300ebba7d3cee:1
Aug 17 12:54:42.228 [INFO] [name:forwarder-ovs-jck5d] (32)                                ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuClient.Request() span=03e9f52b95487f7f:562cd69da38879ba:0ff81ec2ba0aa6bf:1
Aug 17 12:54:42.228 [INFO] [name:forwarder-ovs-jck5d] (33)                                 ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsClient.Request() span=03e9f52b95487f7f:089e439bf8c15434:562cd69da38879ba:1
Aug 17 12:54:42.229 [INFO] [name:forwarder-ovs-jck5d] (34)                                  ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesClient.Request() span=03e9f52b95487f7f:0ef229f0482264e0:089e439bf8c15434:1
Aug 17 12:54:42.229 [INFO] [name:forwarder-ovs-jck5d] (35)                                   ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressClient.Request() span=03e9f52b95487f7f:5e3a1e0835a4996e:0ef229f0482264e0:1
Aug 17 12:54:42.229 [INFO] [name:forwarder-ovs-jck5d] (36)                                    ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectClient.Request() span=03e9f52b95487f7f:40d6a179174cf83f:5e3a1e0835a4996e:1
Aug 17 12:54:42.229 [INFO] [name:forwarder-ovs-jck5d] (37)                                     ⎆ sdk-ovs/pkg/networkservice/mechanisms/kernel/kernelClient.Request() span=03e9f52b95487f7f:4b0811a7a2465bff:40d6a179174cf83f:1
Aug 17 12:54:42.229 [INFO] [name:forwarder-ovs-jck5d] (37.1)                                       request-diff={"mechanism_preferences":{"+0":{"cls":"LOCAL","type":"KERNEL"}}} span=03e9f52b95487f7f:4b0811a7a2465bff:40d6a179174cf83f:1
Aug 17 12:54:42.229 [INFO] [name:forwarder-ovs-jck5d] (38)                                      ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=03e9f52b95487f7f:155b9120674b1516:4b0811a7a2465bff:1
Aug 17 12:54:42.230 [INFO] [name:forwarder-ovs-jck5d] (39)                                       ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanClient.Request() span=03e9f52b95487f7f:0dfe92a2ba8c3001:155b9120674b1516:1
Aug 17 12:54:42.230 [INFO] [name:forwarder-ovs-jck5d] (39.1)                                         request-diff={"mechanism_preferences":{"+1":{"cls":"REMOTE","type":"VXLAN"}}} span=03e9f52b95487f7f:0dfe92a2ba8c3001:155b9120674b1516:1
Aug 17 12:54:42.230 [INFO] [name:forwarder-ovs-jck5d] (40)                                        ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniClient.Request() span=03e9f52b95487f7f:7086d6c41d5b7ad4:0dfe92a2ba8c3001:1
Aug 17 12:54:42.230 [INFO] [name:forwarder-ovs-jck5d] (40.1)                                          request-diff={"mechanism_preferences":{"1":{"parameters":{"+src_ip":"10.10.10.33","+src_port":"4789"}}}} span=03e9f52b95487f7f:7086d6c41d5b7ad4:0dfe92a2ba8c3001:1
Aug 17 12:54:42.230 [INFO] [name:forwarder-ovs-jck5d] (41)                                         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request() span=03e9f52b95487f7f:5500bc4a788a7161:7086d6c41d5b7ad4:1
Aug 17 12:54:42.230 [INFO] [name:forwarder-ovs-jck5d] (42)                                          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request() span=03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
Aug 17 12:54:42.231 [INFO] [name:forwarder-ovs-jck5d] (42.1)                                            request-diff={"connection":{"mechanism":{"parameters":{"inodeURL":"inode://3/4026535837"}}}} span=03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
Aug 17 12:54:42.231 [INFO] [name:forwarder-ovs-jck5d] (43)                                           ⎆ sdk/pkg/networkservice/common/null/nullClient.Request() span=03e9f52b95487f7f:164d524467bbb39d:48252471ad377f27:1
Aug 17 12:54:42.231 [INFO] [name:forwarder-ovs-jck5d] (44)                                            ⎆ api/pkg/api/networkservice/networkServiceClient.Request() span=03e9f52b95487f7f:5eebd95730e3f5c3:164d524467bbb39d:1
Aug 17 12:54:42.403 [INFO] [name:forwarder-ovs-jck5d] (44.1)                                              response={"id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","network_service":"icmp-responder","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://3/4026535837","name":"icmp-respo-356b"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":5,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.0uXakzic0NC44YVPyyDA_32jy9qQrIShPT5E5Hva0KWZOSBts0FthOtLu7JcuiHrQ_XQr30oF7ByUrpvm4s48Q","expires":{"seconds":1629205482,"nanos":103258984}},{"name":"forwarder-ovs-vc7qb","id":"df936088-582e-41db-8205-32e8d8b849b6","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.YRK2sbi52Tk7OfIGhocjZrHq_nOAXlgCg9-tceigiyK5bsAHMAe_1oMBmZswke2eNVpMn1bcF7mdHUzMH3PuHg","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"5be19c57-bff5-467d-be43-6eb2853c83e2","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.UlrMLPTy01M7Bjbded8Fr1O-hRtwa1-bweunvEfZInYy_fGxBx37Osrwvw42Zckz4lSpkS-JLuKzdJTdWBm7tg","expires":{"seconds":1629205482,"nanos":205443660}},{"name":"nsmgr-8hs7f","id":"a5052de1-cb38-44c0-ae11-edb4730ae064","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.il2Xax2TBZ6CATGWNuQvF4D5O-nlv7LYTkfhw8tzl-lShp3OM-J5qVk0mmzJ5_Xfkko-lNrGakfI3nf0GTYrvQ","expires":{"seconds":1629205482,"nanos":217344406}},{"name":"forwarder-ovs-jck5d","id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.C5l2Q_DY_ttt-IIiQHBfbfZKwnsZ1bKhj5Cb--lYac9oz-TvCLMUpDG_x4QHJ-NJOG4jx8LaZR6b_paxcxMc1w","expires":{"seconds":1629207419}},{"name":"nsmgr-8hs7f","id":"fbcd7003-f6ad-471b-9bad-f212e54566cd","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MjA1NDgyLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.s-zXOGagdPesvWqpJcHp_3TnFMLEJv4dqcziSgSKYl4r1-I9n-L5DF9b9ViDbtyUMPNiTCNZqoP-G3c3-S20PQ","expires":{"seconds":1629205482,"nanos":301924123}},{"name":"nse-kernel-6444cdcc55-jjjfv","id":"356b02b4-7bc0-45fe-86e2-70ccd4358632","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.QE09R28E5dsyC7t95845MkkvvBRNb04NfSN0kXm1ITfFCCwJLANvbJY5MpeJWW52J43b5T3YzMu5LSi7dMMq9Q","expires":{"seconds":1629205482,"nanos":305529376}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"} span=03e9f52b95487f7f:5eebd95730e3f5c3:164d524467bbb39d:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5eebd95730e3f5c3:164d524467bbb39d:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:164d524467bbb39d:48252471ad377f27:1
Aug 17 12:54:42.403 [INFO] [name:forwarder-ovs-jck5d] (42.2)                                            response-diff={"mechanism":{"parameters":{"inodeURL":"file:///proc/51245/fd/12"}}} span=03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5500bc4a788a7161:7086d6c41d5b7ad4:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7086d6c41d5b7ad4:0dfe92a2ba8c3001:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0dfe92a2ba8c3001:155b9120674b1516:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:155b9120674b1516:4b0811a7a2465bff:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:4b0811a7a2465bff:40d6a179174cf83f:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:40d6a179174cf83f:5e3a1e0835a4996e:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5e3a1e0835a4996e:0ef229f0482264e0:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0ef229f0482264e0:089e439bf8c15434:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:089e439bf8c15434:562cd69da38879ba:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:562cd69da38879ba:0ff81ec2ba0aa6bf:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0ff81ec2ba0aa6bf:7cd300ebba7d3cee:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7cd300ebba7d3cee:3fbd6ab211997370:1
Aug 17 12:54:42.406 [INFO] [name:forwarder-ovs-jck5d] (29.2)                               response-diff={"mechanism":{"cls":"REMOTE","parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+src_ip":"10.10.10.22","+src_port":"4789","+vni":"9832580","-inodeURL":"file:///proc/51245/fd/12","-name":"icmp-respo-356b"},"type":"VXLAN"}} span=03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:2eee0b6eb144c851:3b56478b9a5c05c8:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:3b56478b9a5c05c8:19fb894ed9f7f635:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:19fb894ed9f7f635:541f71b4a17c3c8f:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:541f71b4a17c3c8f:7e05a0d199774e24:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7e05a0d199774e24:527ba635996fad7d:1
Aug 17 12:54:42.407 [INFO] [name:forwarder-ovs-jck5d] (23.1)                         request-diff={"connection":{"mechanism":{"cls":"REMOTE","parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+src_ip":"10.10.10.22","+src_port":"4789","+vni":"9832580","-inodeURL":"inode://3/4026535837","-name":"icmp-respo-356b"},"type":"VXLAN"},"path":{"path_segments":{"5":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.C5l2Q_DY_ttt-IIiQHBfbfZKwnsZ1bKhj5Cb--lYac9oz-TvCLMUpDG_x4QHJ-NJOG4jx8LaZR6b_paxcxMc1w"},"6":{"expires":{"nanos":301924123,"seconds":1629205482},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MjA1NDgyLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.s-zXOGagdPesvWqpJcHp_3TnFMLEJv4dqcziSgSKYl4r1-I9n-L5DF9b9ViDbtyUMPNiTCNZqoP-G3c3-S20PQ"},"7":{"expires":{"nanos":305529376,"seconds":1629205482},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.QE09R28E5dsyC7t95845MkkvvBRNb04NfSN0kXm1ITfFCCwJLANvbJY5MpeJWW52J43b5T3YzMu5LSi7dMMq9Q"}}}},"mechanism_preferences":{"-1":{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.33","src_port":"4789"}},"0":{"cls":"REMOTE","parameters":{"+src_ip":"10.10.10.22","+src_port":"4789"},"type":"VXLAN"}}} span=03e9f52b95487f7f:527ba635996fad7d:0e18efecba32b8ea:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:527ba635996fad7d:0e18efecba32b8ea:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0e18efecba32b8ea:08776bc38c65c580:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:08776bc38c65c580:584d665bbbbc6573:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:584d665bbbbc6573:09678d30a096de49:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:09678d30a096de49:4cfad252feee66ca:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:4cfad252feee66ca:63d6a7283737ab3c:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:63d6a7283737ab3c:720a4aff192ac769:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:720a4aff192ac769:6365ab165c9d2342:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:6365ab165c9d2342:2a271d7f7444fb04:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:2a271d7f7444fb04:25468e3012761c18:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:25468e3012761c18:5fff97c6694d58a1:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5fff97c6694d58a1:12753f2931c02cb3:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:12753f2931c02cb3:04996bd5df684e5c:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:04996bd5df684e5c:303a460c70ffe464:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:303a460c70ffe464:209c9732b7709b77:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:209c9732b7709b77:485fb6841c630633:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:485fb6841c630633:7190c8ee82847f23:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7190c8ee82847f23:6ee58b63ac81944e:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:6ee58b63ac81944e:76f00b136766d430:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:76f00b136766d430:609c63bb3c8de1a7:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:609c63bb3c8de1a7:3d9c60d29dff250f:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:3d9c60d29dff250f:7a7cce123fb62752:1
Aug 17 12:54:42.411 [INFO] [name:forwarder-ovs-jck5d] (1.3)   response-diff={"id":"a5052de1-cb38-44c0-ae11-edb4730ae064","path":{"index":4}} span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:25e1e78f96c18120:66a47f186fd3a3c9:1
2021/08/17 12:54:45 Reporting span 7adb6f3bf789e1b8:5204e92bfe118ef2:6803c6dd3d17691d:1
Aug 17 12:54:46.118 [INFO] [name:forwarder-ovs-jck5d] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (1.1)   request={"connection":{"id":"e43e7390-fb55-467a-a3c7-1dc1a338c644","network_service":"icmp-responder","context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODUsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.OKq5NDUFeRtTLp--fJKxJjosBFigpvJNwjLPRTWbpZxcRav2FAgERXRVxTe9LlYcjn59omnI2pM9IkLwfhaJGA","expires":{"seconds":1629205485,"nanos":912920715}},{"name":"forwarder-ovs-vc7qb","id":"18bd5275-1853-4e09-9ad6-929771551ec5","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.-syXmY0abGHOc8BNLDz82Kkn3zo4yWB3qrf1AV4kAyf7pAdAxsj9otAxDCSy5l77AdiTsvLjBR6_52bc6sRQgQ","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"98e62743-eaae-48ca-ac89-175813a316db","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODYsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.EmajpYu5XjSh4jBIklQhKuNIkmTIPPryDVDtFLz-xSHCkrmaA8VnfhV-5Lud7vg38rGC1eakgc45Xja0oeTpsg","expires":{"seconds":1629205486,"nanos":107575748}},{"name":"nsmgr-8hs7f","id":"e43e7390-fb55-467a-a3c7-1dc1a338c644","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODYsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.wcBkDpU9T5R0_IEPnoHe2lP00YfCnpJlYige_GY4bFo8MZ3zi5HWWrfTbLUdI1cvHiYZJUve4LOTjdG_Psek1A","expires":{"seconds":1629205486,"nanos":109801924}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (1.2)   request-diff={"connection":{"id":"b128d4d5-6d06-4842-8f01-226e27d5fa92","path":{"index":5,"path_segments":{"+5":{"name":"forwarder-ovs-jck5d","id":"b128d4d5-6d06-4842-8f01-226e27d5fa92"},"4":{"expires":{"nanos":118220347},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODYsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.J483oxn2_J8Q1aKsAQxqfiQ4SXweSIm2q5Y3BxdQ9YeniSVRxxH3skthnlk4EoC6Jhn5bzfpHYSFEtZTi7wtqw"}}}}} span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (2)  ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=4db0233badb1acd3:509094a0ffc76466:30badedf05149b36:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (3)   ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=4db0233badb1acd3:5c1c10f40b476b08:509094a0ffc76466:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (4)    ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=4db0233badb1acd3:3e37c3a7f3af12e5:5c1c10f40b476b08:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (5)     ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=4db0233badb1acd3:6b2367eafb964e45:3e37c3a7f3af12e5:1
Aug 17 12:54:46.119 [INFO] [name:forwarder-ovs-jck5d] (6)      ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=4db0233badb1acd3:73299f1cb4d4cf6e:6b2367eafb964e45:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (7)       ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (7.1)         request-diff={"connection":{"path":{"path_segments":{"5":{"expires":{"seconds":1629207419},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.oZ4hn9Dkoakhn31vELixi-vxITRRT7TworlZ6UBlRg8kl9vFFc1zE4jYwwjGsFEXTUKQ-U0G1sdJx9AE4VNTpw"}}}}} span=4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (8)        ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=4db0233badb1acd3:6dc82ce73e7c9687:0b9d560c035ab0ba:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (9)         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=4db0233badb1acd3:5d0c719d14f3e756:6dc82ce73e7c9687:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=4db0233badb1acd3:5478a5aa3b554c33:5d0c719d14f3e756:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (11)           ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=4db0233badb1acd3:514ef3a009bd68a0:5478a5aa3b554c33:1
Aug 17 12:54:46.120 [INFO] [name:forwarder-ovs-jck5d] (12)            ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=4db0233badb1acd3:5c4cf9be59362cd2:514ef3a009bd68a0:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=4db0233badb1acd3:2d18dd97d11ad99e:5c4cf9be59362cd2:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] (14)              ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] (14.1)                request-diff={"connection":{"mechanism":{"cls":"REMOTE","parameters":{"+src_ip":"10.10.10.22","+src_port":"4789"},"type":"VXLAN"}}} span=4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] (15)               ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] (15.1)                 request-diff={"connection":{"mechanism":{"parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+vni":"9070652"}}}} span=4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] (16)                ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.1)                  connection not established, calling add span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.2)                  inside add method span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.3)                  mech is vxlan span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.4)                  ovs tunnel name: v10101022 span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.125 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.5)                  ovs tunnel port number: 1 span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.125 [INFO] [name:forwarder-ovs-jck5d] (17)                 ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=4db0233badb1acd3:6cb11cfe293b337a:18731484d8cb069a:1
Aug 17 12:54:46.125 [INFO] [name:forwarder-ovs-jck5d] (18)                  ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=4db0233badb1acd3:3a39ddd59c3543a1:6cb11cfe293b337a:1
Aug 17 12:54:46.125 [INFO] [name:forwarder-ovs-jck5d] (19)                   ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=4db0233badb1acd3:1e238d9ec9e46614:3a39ddd59c3543a1:1
Aug 17 12:54:46.125 [INFO] [name:forwarder-ovs-jck5d] (20)                    ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=4db0233badb1acd3:0e1ad8e73b5b3040:1e238d9ec9e46614:1
Aug 17 12:54:46.126 [INFO] [name:forwarder-ovs-jck5d] (21)                     ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=4db0233badb1acd3:2634afd39f918e0a:0e1ad8e73b5b3040:1
Aug 17 12:54:46.126 [INFO] [name:forwarder-ovs-jck5d] (22)                      ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
Aug 17 12:54:46.127 [INFO] [name:forwarder-ovs-jck5d] (23)                       ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=4db0233badb1acd3:40c0eac4ee59079b:6cbfe09b01f3e371:1
Aug 17 12:54:46.130 [ERRO] [name:forwarder-ovs-jck5d] (23.1)                         context canceled
Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:60
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress.(*ipaddressServer).Request
    /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/server.go:64
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes.(*routesServer).Request
    /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors.(*ipNeighborsServer).Request
    /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/server.go:37
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu.(*mtuServer).Request
    /build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/server.go:69 span=4db0233badb1acd3:40c0eac4ee59079b:6cbfe09b01f3e371:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:40c0eac4ee59079b:6cbfe09b01f3e371:1
Aug 17 12:54:46.130 [INFO] [name:forwarder-ovs-jck5d] (22.1)                        response=null span=4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
Aug 17 12:54:46.131 [ERRO] [name:forwarder-ovs-jck5d] (22.2)                        Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
Aug 17 12:54:46.131 [ERRO] [name:forwarder-ovs-jck5d] (21.1)                       Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:2634afd39f918e0a:0e1ad8e73b5b3040:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:2634afd39f918e0a:0e1ad8e73b5b3040:1
Aug 17 12:54:46.132 [ERRO] [name:forwarder-ovs-jck5d] (20.1)                      Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:0e1ad8e73b5b3040:1e238d9ec9e46614:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:0e1ad8e73b5b3040:1e238d9ec9e46614:1
Aug 17 12:54:46.133 [ERRO] [name:forwarder-ovs-jck5d] (19.1)                     Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:1e238d9ec9e46614:3a39ddd59c3543a1:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:1e238d9ec9e46614:3a39ddd59c3543a1:1
Aug 17 12:54:46.134 [ERRO] [name:forwarder-ovs-jck5d] (18.1)                    Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:3a39ddd59c3543a1:6cb11cfe293b337a:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:3a39ddd59c3543a1:6cb11cfe293b337a:1
Aug 17 12:54:46.135 [ERRO] [name:forwarder-ovs-jck5d] (17.1)                   Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:6cb11cfe293b337a:18731484d8cb069a:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6cb11cfe293b337a:18731484d8cb069a:1
Aug 17 12:54:46.135 [ERRO] [name:forwarder-ovs-jck5d] (16.6)                  Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.136 [ERRO] [name:forwarder-ovs-jck5d] (15.2)                 Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
Aug 17 12:54:46.137 [ERRO] [name:forwarder-ovs-jck5d] (14.2)                Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
Aug 17 12:54:46.137 [ERRO] [name:forwarder-ovs-jck5d] (13.1)               Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:2d18dd97d11ad99e:5c4cf9be59362cd2:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:2d18dd97d11ad99e:5c4cf9be59362cd2:1
Aug 17 12:54:46.138 [ERRO] [name:forwarder-ovs-jck5d] (12.1)              Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5c4cf9be59362cd2:514ef3a009bd68a0:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5c4cf9be59362cd2:514ef3a009bd68a0:1
Aug 17 12:54:46.138 [ERRO] [name:forwarder-ovs-jck5d] (11.2)             Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:514ef3a009bd68a0:5478a5aa3b554c33:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:514ef3a009bd68a0:5478a5aa3b554c33:1
Aug 17 12:54:46.139 [ERRO] [name:forwarder-ovs-jck5d] (10.1)            Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5478a5aa3b554c33:5d0c719d14f3e756:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5478a5aa3b554c33:5d0c719d14f3e756:1
Aug 17 12:54:46.139 [ERRO] [name:forwarder-ovs-jck5d] (9.1)           Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5d0c719d14f3e756:6dc82ce73e7c9687:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5d0c719d14f3e756:6dc82ce73e7c9687:1
Aug 17 12:54:46.139 [ERRO] [name:forwarder-ovs-jck5d] (8.1)          Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:6dc82ce73e7c9687:0b9d560c035ab0ba:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6dc82ce73e7c9687:0b9d560c035ab0ba:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (7.2)         Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (6.1)        Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:73299f1cb4d4cf6e:6b2367eafb964e45:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:73299f1cb4d4cf6e:6b2367eafb964e45:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (5.1)       Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:6b2367eafb964e45:3e37c3a7f3af12e5:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6b2367eafb964e45:3e37c3a7f3af12e5:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (4.1)      Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:3e37c3a7f3af12e5:5c1c10f40b476b08:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:3e37c3a7f3af12e5:5c1c10f40b476b08:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (3.1)     Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5c1c10f40b476b08:509094a0ffc76466:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5c1c10f40b476b08:509094a0ffc76466:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (2.1)    Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:509094a0ffc76466:30badedf05149b36:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:509094a0ffc76466:30badedf05149b36:1
Aug 17 12:54:46.140 [ERRO] [name:forwarder-ovs-jck5d] (1.3)   Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:12746015f0ef5b22:324032673fe775c5:1
2021/08/17 12:54:46 Reporting span 7adb6f3bf789e1b8:4814765c13ec3c96:2a5db85f96e876c5:1
Aug 17 12:54:47.810 [INFO] [name:forwarder-ovs-jck5d] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span=33171ea980340c55:3d7f562b904c926f:4082ea13a7356c43:1
Aug 17 12:54:47.810 [INFO] [name:forwarder-ovs-jck5d] (1.1)   request={"connection":{"id":"6bc38cb1-39ef-4cfb-93ca-1e840236ad90","network_service":"icmp-responder","context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.2uj7IrFsA2UgOHfEf0jLwIT27dadMx-Drj7WoIiXJ3yraFc6OhqfOc9NredYIpeW5Staxx4FJ5WtJW4oO07Elw","expires":{"seconds":1629205487,"nanos":403397355}},{"name":"forwarder-ovs-vc7qb","id":"fb061d27-f765-45e1-acc4-da8334770b57","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.VLYeMnlma2vMDSNYDclUOpN9YKsNZtTA8VpHJkbG3cD4DBiUMdPS5yi7hnj8VRsnNJ6hTqBYkCfVs9HjnJW0aw","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"58856609-3808-4045-a5c7-ec4be63fe53e","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.rnxn8bx1zTBEAKNHixiFlPvGOlKcIICj1gQWhMEzqJ7pzkP002psWXOHk6OnuUHreFokYlk5NnvfUv4XZtgOZg","expires":{"seconds":1629205487,"nanos":707657276}},{"name":"nsmgr-8hs7f","id":"6bc38cb1-39ef-4cfb-93ca-1e840236ad90","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.W1DcrnJ6gHpD60SAydPBlpNEDpU1pN-vk4myhgHB7HVrChXhYBTdW61cbRaT8SHZ4q7y9BE5Bqj9u-nJ71Q6Vg","expires":{"seconds":1629205487,"nanos":803101542}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=33171ea980340c55:3d7f562b904c926f:4082ea13a7356c43:1
Aug 17 12:54:47.810 [INFO] [name:forwarder-ovs-jck5d] (1.2)   request-diff={"connection":{"id":"4d8cd1ff-e0ca-43a7-8d11-d5c2786b4339","path":{"index":5,"path_segments":{"+5":{"name":"forwarder-ovs-jck5d","id":"4d8cd1ff-e0ca-43a7-8d11-d5c2786b4339"},"4":{"expires":{"nanos":810141575},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.cKDOhHFqUHeHExw7bgCcJXAoVgOv4iIjMTF3o8stEC-vpUks7yrLr0vpfDkbjb93QVR3Dd0K4tJPHkojr-U4Ig"}}}}} span=33171ea980340c55:3d7f562b904c926f:4082ea13a7356c43:1
Aug 17 12:54:47.811 [INFO] [name:forwarder-ovs-jck5d] (2)  ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=33171ea980340c55:32e55e6ab5639b47:3d7f562b904c926f:1
Aug 17 12:54:47.811 [INFO] [name:forwarder-ovs-jck5d] (3)   ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=33171ea980340c55:444982e0184ab51c:32e55e6ab5639b47:1
Aug 17 12:54:47.811 [INFO] [name:forwarder-ovs-jck5d] (4)    ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=33171ea980340c55:17f3af5424e75d46:444982e0184ab51c:1
Aug 17 12:54:47.811 [INFO] [name:forwarder-ovs-jck5d] (5)     ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=33171ea980340c55:0e18be1c08053317:17f3af5424e75d46:1
Aug 17 12:54:47.811 [INFO] [name:forwarder-ovs-jck5d] (6)      ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=33171ea980340c55:3196cd10c66043c0:0e18be1c08053317:1
Aug 17 12:54:47.811 [INFO] [name:forwarder-ovs-jck5d] (7)       ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=33171ea980340c55:5f159c641dd77dab:3196cd10c66043c0:1
Aug 17 12:54:47.812 [INFO] [name:forwarder-ovs-jck5d] (7.1)         request-diff={"connection":{"path":{"path_segments":{"5":{"expires":{"seconds":1629207419},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.hsPYSiqGiEBiDT-0pITG2S8ISnyDfj8cipLtlcWfw8JaAu2FFJkQTH_qSBanSVXLJGpd-dOd7V341HjYHqUfAw"}}}}} span=33171ea980340c55:5f159c641dd77dab:3196cd10c66043c0:1
Aug 17 12:54:47.812 [INFO] [name:forwarder-ovs-jck5d] (8)        ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=33171ea980340c55:0fa9d5dee39ba2e4:5f159c641dd77dab:1
Aug 17 12:54:47.812 [INFO] [name:forwarder-ovs-jck5d] (9)         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=33171ea980340c55:0b846146600577ad:0fa9d5dee39ba2e4:1
Aug 17 12:54:47.812 [INFO] [name:forwarder-ovs-jck5d] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=33171ea980340c55:7d03570f9b5cbeac:0b846146600577ad:1
Aug 17 12:54:47.812 [INFO] [name:forwarder-ovs-jck5d] (11)           ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=33171ea980340c55:6a59f03cba01225d:7d03570f9b5cbeac:1
Aug 17 12:54:47.812 [INFO] [name:forwarder-ovs-jck5d] (12)            ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=33171ea980340c55:57d1cd41864aa672:6a59f03cba01225d:1
Aug 17 12:54:47.813 [INFO] [name:forwarder-ovs-jck5d] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=33171ea980340c55:1251af30344580fc:57d1cd41864aa672:1
Aug 17 12:54:47.813 [INFO] [name:forwarder-ovs-jck5d] (14)              ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=33171ea980340c55:3f42381edf9545b1:1251af30344580fc:1
Aug 17 12:54:47.814 [INFO] [name:forwarder-ovs-jck5d] (14.1)                request-diff={"connection":{"mechanism":{"cls":"REMOTE","parameters":{"+src_ip":"10.10.10.22","+src_port":"4789"},"type":"VXLAN"}}} span=33171ea980340c55:3f42381edf9545b1:1251af30344580fc:1
Aug 17 12:54:47.814 [INFO] [name:forwarder-ovs-jck5d] (15)               ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=33171ea980340c55:1b8c28d57904c3ae:3f42381edf9545b1:1
Aug 17 12:54:47.814 [INFO] [name:forwarder-ovs-jck5d] (15.1)                 request-diff={"connection":{"mechanism":{"parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+vni":"4273948"}}}} span=33171ea980340c55:1b8c28d57904c3ae:3f42381edf9545b1:1
Aug 17 12:54:47.814 [INFO] [name:forwarder-ovs-jck5d] (16)                ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.1)                  connection not established, calling add span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.2)                  inside add method span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.3)                  mech is vxlan span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.4)                  ovs tunnel name: v10101022 span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.820 [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] (16.5)                  ovs tunnel port number: 1 span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.820 [INFO] [name:forwarder-ovs-jck5d] (17)                 ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=33171ea980340c55:313a64fee9b5e6dd:3799f63ec171df92:1
Aug 17 12:54:47.820 [INFO] [name:forwarder-ovs-jck5d] (18)                  ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=33171ea980340c55:334ae01a91c6bcd3:313a64fee9b5e6dd:1
Aug 17 12:54:47.820 [INFO] [name:forwarder-ovs-jck5d] (19)                   ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=33171ea980340c55:18348afba4d1f762:334ae01a91c6bcd3:1
Aug 17 12:54:47.820 [INFO] [name:forwarder-ovs-jck5d] (20)                    ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=33171ea980340c55:59055e4e99dbac64:18348afba4d1f762:1
Aug 17 12:54:47.820 [INFO] [name:forwarder-ovs-jck5d] (21)                     ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=33171ea980340c55:13ff63e6a1848755:59055e4e99dbac64:1
Aug 17 12:54:47.821 [INFO] [name:forwarder-ovs-jck5d] (22)                      ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=33171ea980340c55:1341f03843aa6ff3:13ff63e6a1848755:1
Aug 17 12:54:47.821 [INFO] [name:forwarder-ovs-jck5d] (23)                       ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=33171ea980340c55:14a9c468f8fe2450:1341f03843aa6ff3:1
2021/08/17 12:54:47 [DEBUG] passed clientURL: unix:///var/lib/networkservicemesh/nsm.io.sock
Aug 17 12:54:47.824 [INFO] [name:forwarder-ovs-jck5d] (24)                        ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request() span=33171ea980340c55:44514a22ea533702:14a9c468f8fe2450:1
Aug 17 12:54:47.824 [INFO] [name:forwarder-ovs-jck5d] (25)                         ⎆ sdk/pkg/networkservice/common/serialize/serializeClient.Request() span=33171ea980340c55:2f94fab5a51a35e3:44514a22ea533702:1
Aug 17 12:54:47.824 [INFO] [name:forwarder-ovs-jck5d] (26)                          ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request() span=33171ea980340c55:40e4f68002ce365f:2f94fab5a51a35e3:1
Aug 17 12:54:47.824 [INFO] [name:forwarder-ovs-jck5d] (27)                           ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request() span=33171ea980340c55:025c6799c9f4c167:40e4f68002ce365f:1
Aug 17 12:54:47.824 [INFO] [name:forwarder-ovs-jck5d] (28)                            ⎆ sdk/pkg/networkservice/common/heal/healClient.Request() span=33171ea980340c55:66522b1ac5378601:025c6799c9f4c167:1
Aug 17 12:54:47.825 [INFO] [name:forwarder-ovs-jck5d] (29)                             ⎆ sdk/pkg/networkservice/common/mechanismtranslation/mechanismTranslationClient.Request() span=33171ea980340c55:4031871f635f2e70:66522b1ac5378601:1
Aug 17 12:54:47.825 [INFO] [name:forwarder-ovs-jck5d] (29.1)                               request-diff={"connection":{"mechanism":{"cls":"","parameters":{"-dst_ip":"10.10.10.33","-dst_port":"4789","-src_ip":"10.10.10.22","-src_port":"4789","-vni":"4273948"},"type":""}},"mechanism_preferences":{"-0":{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}}} span=33171ea980340c55:4031871f635f2e70:66522b1ac5378601:1
Aug 17 12:54:47.825 [INFO] [name:forwarder-ovs-jck5d] (30)                              ⎆ sdk-ovs/pkg/networkservice/l2ovsconnect/l2ConnectClient.Request() span=33171ea980340c55:37df39c6dfb0b273:4031871f635f2e70:1
Aug 17 12:54:47.825 [INFO] [name:forwarder-ovs-jck5d] (31)                               ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=33171ea980340c55:6a6b938b0aa13cac:37df39c6dfb0b273:1
Aug 17 12:54:47.825 [INFO] [name:forwarder-ovs-jck5d] (32)                                ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuClient.Request() span=33171ea980340c55:18d89b93e3c6e52d:6a6b938b0aa13cac:1
Aug 17 12:54:47.825 [INFO] [name:forwarder-ovs-jck5d] (33)                                 ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsClient.Request() span=33171ea980340c55:0d89ae6f2c30831e:18d89b93e3c6e52d:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (34)                                  ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesClient.Request() span=33171ea980340c55:0074594f9572df25:0d89ae6f2c30831e:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (35)                                   ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressClient.Request() span=33171ea980340c55:171e7a38568f99a3:0074594f9572df25:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (36)                                    ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectClient.Request() span=33171ea980340c55:3e31d42098a89dc3:171e7a38568f99a3:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (37)                                     ⎆ sdk-ovs/pkg/networkservice/mechanisms/kernel/kernelClient.Request() span=33171ea980340c55:4576ebaf4460b8ea:3e31d42098a89dc3:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (37.1)                                       request-diff={"mechanism_preferences":{"+0":{"cls":"LOCAL","type":"KERNEL"}}} span=33171ea980340c55:4576ebaf4460b8ea:3e31d42098a89dc3:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (38)                                      ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=33171ea980340c55:7fe9db03dd431cf1:4576ebaf4460b8ea:1
Aug 17 12:54:47.826 [INFO] [name:forwarder-ovs-jck5d] (39)                                       ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanClient.Request() span=33171ea980340c55:10ccfd079b8fc35f:7fe9db03dd431cf1:1
Aug 17 12:54:47.827 [INFO] [name:forwarder-ovs-jck5d] (39.1)                                         request-diff={"mechanism_preferences":{"+1":{"cls":"REMOTE","type":"VXLAN"}}} span=33171ea980340c55:10ccfd079b8fc35f:7fe9db03dd431cf1:1
Aug 17 12:54:47.827 [INFO] [name:forwarder-ovs-jck5d] (40)                                        ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniClient.Request() span=33171ea980340c55:30eb541b70728ea6:10ccfd079b8fc35f:1
Aug 17 12:54:47.827 [INFO] [name:forwarder-ovs-jck5d] (40.1)                                          request-diff={"mechanism_preferences":{"1":{"parameters":{"+src_ip":"10.10.10.33","+src_port":"4789"}}}} span=33171ea980340c55:30eb541b70728ea6:10ccfd079b8fc35f:1
Aug 17 12:54:47.827 [INFO] [name:forwarder-ovs-jck5d] (41)                                         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request() span=33171ea980340c55:4bbde622745cab01:30eb541b70728ea6:1
Aug 17 12:54:47.827 [INFO] [name:forwarder-ovs-jck5d] (42)                                          ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request() span=33171ea980340c55:5622e789510d63da:4bbde622745cab01:1
Aug 17 12:54:47.827 [INFO] [name:forwarder-ovs-jck5d] (43)                                           ⎆ sdk/pkg/networkservice/common/null/nullClient.Request() span=33171ea980340c55:2c79c16386cda3ee:5622e789510d63da:1
Aug 17 12:54:47.828 [INFO] [name:forwarder-ovs-jck5d] (44)                                            ⎆ api/pkg/api/networkservice/networkServiceClient.Request() span=33171ea980340c55:037ec6a58d9fb28b:2c79c16386cda3ee:1
Aug 17 12:54:48.009 [ERRO] [name:forwarder-ovs-jck5d] (44.1)                                              rpc error: code = Unknown desc = Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/ipam/point2pointipam/ipamServer.Request: IPPool is empty
Error returned from api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:62
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/null.(*nullClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/common/null/client.go:43
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/common/mechanisms/sendfd/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.(*recvFDClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/common/mechanisms/recvfd/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/vxlan/vni.(*vniClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/common/mechanisms/vxlan/vni/client.go:53
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
    /go/pkg/mod/github.com/networkservicemesh/sdk@v0.5.1-0.20210810055235-c3154387718f/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk-ovs/pkg/networkservice/mechanisms/vxlan.(*vxlanClient).Request
    /build/local/sdk-ovs/pkg/networkservice/mechanisms/vxlan/client.go:59 span=33171ea980340c55:037ec6a58d9fb28b:2c79c16386cda3ee:1
2021/08/17 12:54:48 Reporting span 33171ea980340c55:037ec6a58d9fb28b:2c79c16386cda3ee:1
Aug 17 12:54:48.009 [INFO] [name:forwarder-ovs-jck5d] (43.1)                                             response=null span=33171ea980340c55:2c79c16386cda3ee:5622e789510d63da:1
pperiyasamy commented 3 years ago

@Bolodya1997 please ignore the above log, looks that happened because of NSM Mgr pod crash. 

Events:
  Type     Reason     Age                   From                Message
  ----     ------     ----                  ----                -------
  Normal   Scheduled  54m                                       Successfully assigned nsm-system/nsmgr-8hs7f to dl380-003
  Warning  Unhealthy  32m                   kubelet, dl380-003  Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "argument list too long": unknown
  Normal   Created    30m (x5 over 54m)     kubelet, dl380-003  Created container nsmgr
  Normal   Pulled     30m (x5 over 54m)     kubelet, dl380-003  Container image "ghcr.io/networkservicemesh/ci/cmd-nsmgr:634a640" already present on machine
  Normal   Started    30m (x5 over 54m)     kubelet, dl380-003  Started container nsmgr
  Warning  Unhealthy  28m (x2 over 35m)     kubelet, dl380-003  Readiness probe failed:
  Warning  Unhealthy  25m (x3 over 34m)     kubelet, dl380-003  Liveness probe failed:
  Warning  Unhealthy  7m15s                 kubelet, dl380-003  Liveness probe failed: failed to initialize tls credentials with spiffe. error=context deadline exceeded
  Warning  Unhealthy  7m15s                 kubelet, dl380-003  Readiness probe errored: rpc error: code = Unknown desc = container not running (eaeb4e6197ef4a15e1deb64074326c4fc9d0ed4fbad1201cacf7d913bb2d22c7)
  Warning  BackOff    5m27s (x60 over 34m)  kubelet, dl380-003  Back-off restarting failed container

would keep you posted if the issue is seen again with latest image..

yuraxdrumz commented 2 years ago

Hey, Seems I have the same issue. Running EKS 1.21 with Calico and VPP without huge pages.

Running a simple setup of 1 nse - 2 nsc's

Everything seems to be working fine, but I see lots of error messages from all the nsm-components as well as healing not working properly (not sure if its related or another issue).

Attached logs per component

forwarder.txt nse.txt nsc.txt nsmgr.txt

EDIT: Seems the errors occur in an interval of 30s-40s. I am guessing this is some kind of refresh mechanism?

EDIT2: I tried to restart nse - it shows a lot of registry errors 1 nsc - has 3 interfaces connected to the nse 2 nsc - has 2 interfaces which are not connected anywhere

Attached logs after nse restart:

nsc2-after-nse-restart.txt nse-after-restart.txt nsc-after-nse-restart.txt

ip addr for nse

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 8981 qdisc mq state UNKNOWN qlen 500
    link/[65534] 
    inet 10.10.99.25/32 scope global eth0
       valid_lft forever preferred_lft forever
3: icmp-respo-2ed1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN qlen 1000
    link/ether 02:fe:65:c5:a7:b2 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.96/32 scope global icmp-respo-2ed1
       valid_lft forever preferred_lft forever
    inet 172.16.1.98/32 scope global icmp-respo-2ed1
       valid_lft forever preferred_lft forever
    inet6 fe80::fe:65ff:fec5:a7b2/64 scope link 
       valid_lft forever preferred_lft forever
4: icmp-respo-e5a3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN qlen 1000
    link/ether 02:fe:a9:44:60:53 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.100/32 scope global icmp-respo-e5a3
       valid_lft forever preferred_lft forever
    inet 172.16.1.98/32 scope global icmp-respo-e5a3
       valid_lft forever preferred_lft forever
    inet 172.16.1.96/32 scope global icmp-respo-e5a3
       valid_lft forever preferred_lft forever
    inet6 fe80::fe:a9ff:fe44:6053/64 scope link 
       valid_lft forever preferred_lft forever

ip addr for nsc 1

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 8981 qdisc mq state UNKNOWN qlen 500
    link/[65534] 
    inet 10.10.158.37/32 scope global eth0
       valid_lft forever preferred_lft forever
34: ovpn-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN qlen 1000
    link/ether 02:fe:b3:bf:db:a4 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.101/32 scope global ovpn-1
       valid_lft forever preferred_lft forever
    inet 172.16.1.99/32 scope global ovpn-1
       valid_lft forever preferred_lft forever
    inet 172.16.1.97/32 scope global ovpn-1
       valid_lft forever preferred_lft forever
    inet6 fe80::fe:b3ff:febf:dba4/64 scope link 
       valid_lft forever preferred_lft forever

ip addr for nsc 2

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 8981 qdisc mq state UNKNOWN group default qlen 500
    link/none 
    inet 10.10.61.79/32 scope global eth0
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 10000
    link/none 
    inet 10.20.30.1/24 scope global tun0
       valid_lft forever preferred_lft forever
26: ovpn-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN group default qlen 1000
    link/ether 02:fe:1c:ac:62:d1 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.97/32 scope global ovpn-1
       valid_lft forever preferred_lft forever
    inet 172.16.1.99/32 scope global ovpn-1
       valid_lft forever preferred_lft forever
    inet6 fe80::fe:1cff:feac:62d1/64 scope link 
       valid_lft forever preferred_lft forever
edwarnicke commented 2 years ago

@yuraxdrumz Which version of NSM are you using?

yuraxdrumz commented 2 years ago

I was running v1.1.1 for a week and then upgraded to v1.2.0 a couple of days ago

edwarnicke commented 2 years ago

@yuraxdrumz And you are seeing the same issue in v1.2.0 after the upgrade?

yuraxdrumz commented 2 years ago

@edwarnicke Yep. I just checked and registry, nsmgr, nsc, nsc-init and forwarder are all on latest images. Same issues.

denis-tingaikin commented 2 years ago

@yuraxdrumz

Canceled desc = context canceled error is expected from registry. In short, each refresh cancels heal and schedules a new one with a new context. I think we can change log level for this message.

Started to look into this.

Q1: Could you point at examples or steps that you're using to reproduce this?

Q2: What do you see in terms of datapath? Are all interfaces correct? Is ping working?

yuraxdrumz commented 2 years ago

  1. Steps

  2. In terms of data path, I see a very strange behavior:

    • If I leave a fresh install with 2 nsc's and 1 nse, everything pings and works as it should for a while
    • If I restart any of the nsc's or nce's, I always see multiple interfaces like in the code I pasted a few comments above and ping works for some interfaces and for some it does not
    • I left a fresh install running over the course of the weekend and I saw the following logs today, I am guessing the IPPOOL error has something to do with why I see the errors and although the interfaces get cleaned up, it seems the ip addresses are still taken, so healing does not work properly. nse-refresh-failed.txt nsmgr-refresh-failed.txt forwarder-refresh-failed.txt nsc-refresh-failed.txt
yuraxdrumz commented 2 years ago

After going through https://github.com/networkservicemesh/deployments-k8s/issues/1999, I see my use case of multiple interfaces with same address is caused by multiple nse's that overwrite each other, so that explains one issue.

edwarnicke commented 2 years ago

@yuraxdrumz What are the remaining issues you are seeing?

edwarnicke commented 2 years ago

@yuraxdrumz What are the remaining issues you are seeing?

caboteria commented 2 years ago

I think that the problem is not that we get "context cancelled" errors but that the code which tries to ignore them isn't working. For example, in the log messages that I get, one of the lines in the backtrace is:

github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryFindClient).Recv;
/go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0/pkg/registry/core/trace/nse_registry.go:56;

...which is https://github.com/networkservicemesh/sdk/blob/v1.5.0/pkg/registry/core/trace/nse_registry.go#L56

The message is rpc error: code = Canceled desc = context canceled; so I think that the log message should be prevented by https://github.com/networkservicemesh/sdk/blob/v1.5.0/pkg/registry/core/trace/nse_registry.go#L53 but the equality comparison is failing so the error gets logged.

https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features indicates that we should be using "errors.Is()" for this purpose instead of "==" so this change might reduce the logspam. I'm just getting into NSM so I haven't set up a dev environment yet but I should be able to take a look in the next couple of weeks unless someone else gets to it sooner.

edwarnicke commented 2 years ago

@caboteria Nice catch! Care to push a PR there? :)