Memcached protocol doesn't work

[yizhengx]

Hi, we implemented memcached protocol in aeraki and used DeathStarBenchmark/hotel-reservation to test.

Meta-protocol-proxy implementation: https://github.com/yizhengx/meta-protocol-proxy/tree/local-queue/src/application_protocols/memcached

We built it using bazel following the instructions in the document.

Example yamls for memcached services

• deployment

  apiVersion: apps/v1
  kind: Deployment
  metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: memcached-rate
  name: memcached-rate
  spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: memcached-rate
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert
        kompose.version: 1.22.0 (955b78124)
        sidecar.istio.io/statsInclusionPrefixes: cluster.outbound,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grp,listener,connection_manager
        sidecar.istio.io/statsInclusionRegexps: http.*
        sidecar.istio.io/bootstrapOverride: aeraki-bootstrap-config
        sidecar.istio.io/proxyImage: yizhengx/aeraki-proxy:rate-limit-cas # custom image we built
      creationTimestamp: null
      labels:
        io.kompose.service: memcached-rate
    spec:
      containers:
        - env:
            - name: MEMCACHED_CACHE_SIZE
              value: "128"
            - name: MEMCACHED_THREADS
              value: "2"
          image: memcached
          name: hotel-reserv-rate-mmc
          ports:
            - containerPort: 11211
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 1000m
      restartPolicy: Always
  status: {}

• service

  apiVersion: v1
  kind: Service
  metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: memcached-rate
  name: memcached-rate
  spec:
  ports:
    - name: tcp-metaprotocol-memcached-memcached-rate
      protocol: TCP
      port: 11211
      targetPort: 11211
  selector:
    io.kompose.service: memcached-rate
  status:
  loadBalancer: {}

I really appreciate any advice on this including what we missed and how we can find the root cause. Thanks!

[zhaohuabing]

@yizhengx Can you paste the Envoy config dump?

[yizhengx]

@yizhengx Can you paste the Envoy config dump?

Hi! I'm not sure what exactly you're looking for, but I ran istioctl proxy-status for one of the memcached proxy:

yizhengx@node0:~/aeraki-benchmark/hotel-res$ istioctl proxy-status memcached-profile-6d4b6b647f-nw6rl.hotel-res
Clusters Match
--- Istiod Listeners
+++ Envoy Listeners
@@ -3233,51 +3233,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-rate.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-rate.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "memcached-rate.hotel-res.svc.cluster.local_11211"
-                                                        },
-                                                "statPrefix": "outbound|11211||memcached-rate.hotel-res.svc.cluster.local"
+                                                            "route_config_name": "memcached-rate.hotel-res.svc.cluster.local_11211"
+                                                        },
+                                                "stat_prefix": "outbound|11211||memcached-rate.hotel-res.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -4991,51 +4991,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-reserve.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-reserve.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "memcached-reserve.hotel-res.svc.cluster.local_11211"
-                                                        },
-                                                "statPrefix": "outbound|11211||memcached-reserve.hotel-res.svc.cluster.local"
+                                                            "route_config_name": "memcached-reserve.hotel-res.svc.cluster.local_11211"
+                                                        },
+                                                "stat_prefix": "outbound|11211||memcached-reserve.hotel-res.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -5889,51 +5889,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "memcached-profile.hotel-res.svc.cluster.local_11211"
-                                                        },
-                                                "statPrefix": "outbound|11211||memcached-profile.hotel-res.svc.cluster.local"
+                                                            "route_config_name": "memcached-profile.hotel-res.svc.cluster.local_11211"
+                                                        },
+                                                "stat_prefix": "outbound|11211||memcached-profile.hotel-res.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -6214,51 +6214,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "thrift",
+                                                "application_protocol": "thrift",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.thrift"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "thrift-sample-server.meta-thrift.svc.cluster.local"
+                                                                                    "destination_service": "thrift-sample-server.meta-thrift.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
-                                                        },
-                                                "statPrefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local"
+                                                            "route_config_name": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
+                                                        },
+                                                "stat_prefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -7170,44 +7170,44 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
-                                                "routeConfig": {
+                                                "route_config": {
                                                             "name": "inbound|11211||",
                                                             "routes": [
                                                                         {
                                                                                     "route": {
                                                                                                 "cluster": "inbound|11211||"
                                                                                             }
                                                                                 }
                                                                     ]
                                                         },
-                                                "statPrefix": "inbound|11211||"
+                                                "stat_prefix": "inbound|11211||"
                                             }
                                     }
                                 }
                             ],
                             "transportSocket": {
                                 "name": "envoy.transport_sockets.tls",
                                 "typedConfig": {
@@ -7314,44 +7314,44 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
-                                                "routeConfig": {
+                                                "route_config": {
                                                             "name": "inbound|11211||",
                                                             "routes": [
                                                                         {
                                                                                     "route": {
                                                                                                 "cluster": "inbound|11211||"
                                                                                             }
                                                                                 }
                                                                     ]
                                                         },
-                                                "statPrefix": "inbound|11211||"
+                                                "stat_prefix": "inbound|11211||"
                                             }
                                     }
                                 }
                             ],
                             "name": "0.0.0.0_11211"
                         },
                         {
@@ -7383,44 +7383,44 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
-                                                "routeConfig": {
+                                                "route_config": {
                                                             "name": "inbound|11211||",
                                                             "routes": [
                                                                         {
                                                                                     "route": {
                                                                                                 "cluster": "inbound|11211||"
                                                                                             }
                                                                                 }
                                                                     ]
                                                         },
-                                                "statPrefix": "inbound|11211||"
+                                                "stat_prefix": "inbound|11211||"
                                             }
                                     }
                                 }
                             ],
                             "name": "0.0.0.0_11211"
                         }
                     ],

Routes Match (RDS last loaded at Tue, 30 Apr 2024 22:41:35 CDT)

Output from running istioctl proxy-config for a specific pod

yizhengx@node0:~/aeraki-benchmark/hotel-res$ istioctl proxy-config all -n hotel-res memcached-profile-6d4b6b647f-nw6rl
Error: error unmarshalling config dump response from Envoy: proto: (line 21366:13): unable to resolve "type.googleapis.com/aeraki.meta_protocol_proxy.admin.v1alpha.RoutesConfigDump": "not found"

[zhaohuabing]

@yizhengx The config dump looks good to me since the original TCP_Filter has been replaced with Meta_protocol_filter and the codec is "name": "aeraki.meta_protocol.codec.memcached".

For the next step, I suggest you turn on trace and look into the log to see how a memchace request is processed and what could be potentially wrong.