search

apache / dubbo-spi-extensions

Apache Dubbo SPI Extensions
https://dubbo.apache.org/
Apache License 2.0
190 stars 112 forks source link

dubbo-xds not support WeightedCluster #271

Open AYue-94 opened 1 year ago

AYue-94 commented 1 year ago

Environment

Steps to reproduce this issue

1）use dubbo-samples/3-extensions/registry/dubbo-samples-xds 2）kubectl label pods -n dubbo-demo {podName1} version=v1 && kubectl label pods -n dubbo-demo {podName2} version=v2 3）kubectl apply -f VirtualService.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: dubbo-samples-xds-provider
  namespace: dubbo-demo
spec:
  hosts:
    - dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
  http:
    - match:
        - port: 50051
      route:
        - destination:
            host: dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
            subset: v1
            port:
              number: 50051
          weight: 50
        - destination:
            host: dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
            subset: v2
            port:
              number: 50051
          weight: 50

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: dubbo-samples-xds-provider
  namespace: dubbo-demo
spec:
  host: dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Expected Behavior

consumer can find 2 invokers.

Actual Behavior

no invokers found.

May 31, 2023 5:47:26 AM org.apache.dubbo.rpc.cluster.directory.AbstractDirectory warn
WARNING:  [DUBBO] No provider available after connectivity filter for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 All validInvokers' size: 0 All routed invokers' size: 0 All invokers' size: 0 from registry istiod.istio-system.svc:15012 on the consumer 10.1.3.76 using the dubbo version 3.2.0., dubbo version: 3.2.0, current host: 10.1.3.76, error code: 2-2. This may be caused by provider server or registry center crashed, go to https://dubbo.apache.org/faq/2/2 to find instructions.
org.apache.dubbo.rpc.RpcException: Failed to invoke the method sayHello in the service org.apache.dubbo.samples.api.GreetingService. No provider available for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 from registry istiod.istio-system.svc:15012 on the consumer 10.1.3.76 using the dubbo version 3.2.0. Please check if the providers have been started and registered.
    at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.checkInvokers(AbstractClusterInvoker.java:367)
    at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:60)
    at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:341)
    at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:101)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:51)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:61)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:41)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:118)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
    at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:194)
    at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:91)
    at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:103)
    at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:131)
    at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:284)
    at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:57)
    at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:75)
    at org.apache.dubbo.samples.api.GreetingServiceDubboProxy0.sayHello(GreetingServiceDubboProxy0.java)
    at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy27.sayHello(Unknown Source)
    at org.apache.dubbo.samples.action.GreetingServiceConsumer.doSayHello(GreetingServiceConsumer.java:34)
    at org.apache.dubbo.samples.ConsumerBootstrap.main(ConsumerBootstrap.java:38)
May 31, 2023 5:47:26 AM org.apache.dubbo.rpc.cluster.SingleRouterChain warn
WARNING:  [DUBBO] No provider available after route for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 from registry 10.1.3.76 on the consumer 10.1.3.76 using the dubbo version 3.2.0. Router snapshot is below:
[ Parent (Input: 0) (Current Node Output: 0) (Chain Node Output: 0) ] Input: Empty -> Chain Node Output: Empty
  [ MockInvokersSelector (Input: 0) (Current Node Output: 0) (Chain Node Output: 0) Router message: Empty invokers. Directly return. ] Current Node Output: Empty, dubbo version: 3.2.0, current host: 10.1.3.76, error code: 2-2. This may be caused by No provider available after route for the service, go to https://dubbo.apache.org/faq/2/2 to find instructions.
May 31, 2023 5:47:26 AM org.apache.dubbo.rpc.cluster.directory.AbstractDirectory warn
WARNING:  [DUBBO] No provider available after connectivity filter for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 All validInvokers' size: 0 All routed invokers' size: 0 All invokers' size: 0 from registry istiod.istio-system.svc:15012 on the consumer 10.1.3.76 using the dubbo version 3.2.0., dubbo version: 3.2.0, current host: 10.1.3.76, error code: 2-2. This may be caused by provider server or registry center crashed, go to https://dubbo.apache.org/faq/2/2 to find instructions.
AYue-94 commented 1 year ago

我大概看了一下，主要有两个地方可能有问题。 1）org.apache.dubbo.registry.xds.util.protocol.impl.RdsProtocol 解析的时候没有解析weightcluster。 image 但是却提供了xdsrouter相关功能，但是实际上底层invokers并不会存在。 2）org.apache.dubbo.rpc.cluster.router.xds.XdsRouter#computeSubset 这里在路由的时候，构造bitlist应该有问题，是不是应该这样做？ image

除了上面两个问题以外，我想问一下proxyless模式现在是不是非生产可用？

AlbumenJ commented 1 year ago

@aamingaa PTAL

aamingaa commented 1 year ago

怎么直接apply virtual.yml了，前面还有好几步呢，参考下下面的命令 mvn clean package -DskipTests -U cd ./dubbo-samples-xds-provider/ docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-provider_0.0.1 . cd ../dubbo-samples-xds-consumer/ docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-consumer_0.0.1 . cd ../ kubectl apply -f ./deploy/Namespace.yml kubens dubbo-demo cd ./dubbo-samples-xds-provider/src/main/resources/k8s kubectl apply -f Deployment.yml kubectl apply -f Deployment-2.yml kubectl apply -f Service.yml kubectl apply -f Virtual-Service.yml cd ../../../../../dubbo-samples-xds-consumer/src/main/resources/k8s kubectl apply -f Deployment.yml kubectl apply -f Service.yml

AYue-94 commented 1 year ago

怎么直接apply virtual.yml了，前面还有好几步呢，参考下下面的命令 mvn clean package -DskipTests -U cd ./dubbo-samples-xds-provider/ docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-provider_0.0.1 . cd ../dubbo-samples-xds-consumer/ docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-consumer_0.0.1 . cd ../ kubectl apply -f ./deploy/Namespace.yml kubens dubbo-demo cd ./dubbo-samples-xds-provider/src/main/resources/k8s kubectl apply -f Deployment.yml kubectl apply -f Deployment-2.yml kubectl apply -f Service.yml kubectl apply -f Virtual-Service.yml cd ../../../../../dubbo-samples-xds-consumer/src/main/resources/k8s kubectl apply -f Deployment.yml kubectl apply -f Service.yml

1）我省略了前面Deployment和Service的部署，这几步肯定要做的。 2）dubbo-samples-xds并没有提供virtualService.yaml，所以我把sample项目部署、打标、VirtualService资源的定义分开描述了。 3）sidecar模式确实没问题，问题是proxyless模式

aamingaa commented 1 year ago

这有一份proxyless下模式的virtualService.yml的配置文件，你对照着试一下？https://github.com/apache/dubbo-samples/compare/master...aamingaa:dubbo-samples:test/xds_debug_3.2.0-beta.7-SNAPSHOT

AYue-94 commented 1 year ago

这有一份proxyless下模式的virtualService.yml的配置文件，你对照着试一下？apache/dubbo-samples@master...aamingaa:dubbo-samples:test/xds_debug_3.2.0-beta.7-SNAPSHOT

@aamingaa

这个我也试了，不行。因为sidecar模式和proxyless模式，除了application.properties中几个配置项不一样以外。其他部署上是没有区别的（除了namespace不打标注入istioproxy以外），所以sidecar能通，proxyless我这边也不会部署有什么问题。

如果我没理解错的话，运行时通过PilotExchanger.getInstance().getEndpoints("dubbo-samples-xds-provider")应该能获取到provider的所有Endpoint，包括v1的和v2的，但是现在也获取不到。

songxiaosheng commented 1 year ago

这周也遇到了同样的异常ProxyLess模式，

image