kmesh-net / kmesh

High Performance ServiceMesh Data Plane Based on Programmable Kernel
https://kmesh.net
Apache License 2.0
464 stars 70 forks source link

waypoint failed to process ipv4-mapped ipv6 connections #756

Closed YaoZengzeng closed 2 months ago

YaoZengzeng commented 3 months ago

What happened:

In default k8s environment (non pure ipv6 env), java applications will use ipv4-mapped ipv6 to create connections by default.

In the absence of waypoint, Kmesh has alreay support this scenario, ref: https://github.com/kmesh-net/kmesh/issues/291

But it still fails when there is a waypoint.

What you expected to happen:

Kmesh could handle both with and without waypoint.

How to reproduce it (as minimally and precisely as possible):

Deploy bookinfo where reviews service is a java application. When it accesses the ratings service, it would create ipv4-mapped ipv6 connection. Then deploy waypoint for svc ratings, ref: https://kmesh.net/en/docs/userguide/install_waypoint/

Access bookinfo and Kmesh daemon will output the following error log:

time="2024-08-20T08:33:55Z" level=info msg="can't find service correspond workload: ratings-svc-waypoint-5d86d48c55-flxrd" subsys=pkg/telemetry
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: origin addr=[10:0a60:4cfe:0000:0000:0000:0000:0000:0000:9080]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[10:0a60:4cfe:0000:0000:0000:0000:0000:0000:9080]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[0a60:c461:0000:0000:0000:0000:0000:0000:15008]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: enter cgroup/connect6\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: origin addr=[10:0a60:c461:0000:0000:0000:0000:0000:0000:15008]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[10:0a60:c461:0000:0000:0000:0000:0000:0000:15008]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[BACKEND] DEBUG: access the backend by service:2317460390\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[BACKEND] DEBUG: get the backend addr=[0af4:01db:0000:0000:0000:0000:0000:0000:15019]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: enter cgroup/connect6\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: origin addr=[10:0af4:01db:0000:0000:0000:0000:0000:0000:15019]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[10:0af4:01db:0000:0000:0000:0000:0000:0000:15019]\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[SOCKOPS] ERR: enable encoding metadata failed!, err is -16" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="[SENDMSG] DEBUG: get valid dst, do encoding...\n" subsys=ebpf
time="2024-08-20T08:33:55Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-20T08:33:55Z" level=info msg="can't find service correspond workload: ratings-svc-waypoint-5d86d48c55-flxrd" subsys=pkg/telemetry
time="2024-08-20T08:33:55Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-20T08:33:55Z" level=info msg="can't find service correspond workload: reviews-svc-waypoint-6884756fc5-wvdvh" subsys=pkg/telemetry
time="2024-08-20T08:33:55Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-20T08:33:55Z" level=info msg="can't find service correspond workload: reviews-svc-waypoint-6884756fc5-wvdvh" subsys=pkg/telemetry

Anything else we need to know?:

Environment:

YaoZengzeng commented 2 months ago
root@kurator-linux-0002:~/waypoint# klp ratings-svc-waypoint-5d86d48c55-flxrd | grep tlv
2024-08-22T06:29:15.683565Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:29       kmesh_tlv: new connection accepted      thread=28
2024-08-22T06:29:15.686203Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:52       kmesh tlv listener filter parse buffer  thread=28
2024-08-22T06:29:15.686206Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:59       already has 28 bytes in the buffer, expected length is 5        thread=28
2024-08-22T06:29:15.686209Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:63       tlv parse state is TypeAndLength        thread=28
2024-08-22T06:29:15.686211Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:66       process TVL_TYPE_SERVICE_ADDRESS        thread=28
2024-08-22T06:29:15.686213Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:71       get tlv length 18       thread=28
2024-08-22T06:29:15.686216Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:59       already has 28 bytes in the buffer, expected length is 23       thread=28
2024-08-22T06:29:15.686218Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:100      tlv parse state is Content      thread=28
2024-08-22T06:29:15.686231Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:128      original dst addresss is [a60:4cfe::]:9080      thread=28
2024-08-22T06:29:15.686270Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:59       already has 28 bytes in the buffer, expected length is 28       thread=28
2024-08-22T06:29:15.686277Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:63       tlv parse state is TypeAndLength        thread=28
2024-08-22T06:29:15.686279Z     trace   envoy filter source/extensions/filters/listener/kmesh_tlv/kmesh_tlv.cc:87       process TLV_TYPE_ENDING thread=28

According to the log of the waypoint shown above, we could see that it did receive the ipv6 connection and parse the target IP content from it.

But its address content is [{rating svc ipv4 addr}::]:9080, which is invalid. We should also wrap it into an ipv4-mapped ipv6 address, like [::FFFF:{rating svc ipv4 addr}]