dubbo注册服务和cloud服务如何在nacos上区分开？

352926 commented 3 years ago

Environment

Spring Cloud Alibaba 2.2.3.RELEASE
Dubbo version: 2.7.8
Java version: 1.8

当前有如下问题： 10.20.2.18:20880为阿里云k8s上建立的私网集群负载均衡服务

容器副本两台：172.17.0.190和172.17.0.191
容器（PodIP：172.17.0.190）无法访问 负载均衡：10.20.2.18:20880（轮询到自身172.17.0.190:20880时）
容器（PodIP：172.17.0.190）可以访问 负载均衡：10.20.2.18:20880（轮询到172.17.0.191:20880时）

容器（PodIP：172.17.0.191）无法访问 负载均衡：10.20.2.18:20880（轮询到自身172.17.0.191:20880时）
容器（PodIP：172.17.0.191）可以访问 负载均衡：10.20.2.18:20880（轮询到172.17.0.190:20880时）

具体现象：任何一台容器消费dubbo时候，一次成功，一次失败。

出现这个问题的原因是，nacos配置直接用负载均衡ip做了注册：

spring.cloud.nacos.discovery.server-addr=xx.xx.xx.xx:8848
spring.cloud.nacos.discovery.ip=10.20.2.18
spring.cloud.nacos.discovery.port=8080

dubbo注册服务时使用PodIP注册(172.17.0.190:20880和172.17.0.191:20880)，消费的时候却使用了负载均衡ip，即10.20.2.18:20880

网上搜了一圈，好像K8S容器自身与负载均衡解析到自身时是不通的。

为什么非要指定负载均衡ip？因为公司环境相对复杂，服务器资源在多个阿里云账号下，虽然拉了内网专线

但不属于同一个K8S内网，因此不同账号下的容器内部Pod网络不通（即172网段），需要做负载均衡转发。

因此想到这种解决方案，如果纯spring cloud的话，没有问题，恰恰dubbo和springcloud注册的是同一个服务名（application.name），且指定了同一个ip：10.20.2.18

理想状态，dubbo 服务纯粹仅供系统内部调用，因此注册使用PodIP即可。而对外暴露feign调用的话，使用负载均衡ip做解析

但是目前就是 dubbo虽然注册上去是172网段，但是消费的时候，全部使用10.20.2.18负载均衡网段。

想来想去，这套架构，最合适的方案就是：系统与系统间feign用负载均衡ip(10.xx)，系统内部dubbo用容器内部网段(172.xx)消费

请问大牛，想要指定dubbo消费网段要怎么做？或者说，dubbo的服务与 cloud的服务区分开，也能间接解决此问题！

Steps to reproduce this issue

将nacos注册ip和端口指定到k8s负载均衡ip上
任意一个consumer消费provider，副本数为2，一次可用，一次不可用
检测的方法还可以进容器，直接 curl http://负载均衡ip:8080/xxx ，一次通，一次不通

Pls. provide [GitHub address] to reproduce this issue.

Expected Result

172.17.0.191 消费 172.17.0.190:20880或172.17.0.191:20880

Actual Result

172.17.0.191 消费 10.20.2.18:20880 一次通，一次拒绝

2021-04-30 14:47:57.689 INFO  [dubbo-client-idleCheck-thread-1] ReconnectTimerTask -  [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [172.17.0.190:0 -> /10.20.2.19:20880]], dubbo version: 2.7.8, current host: 172.17.0.190
2021-04-30 14:47:57.694 ERROR [dubbo-client-idleCheck-thread-1] ReconnectTimerTask -  [DUBBO] Fail to connect to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [172.17.0.190:0 -> /10.20.2.19:20880]], dubbo version: 2.7.8, current host: 172.17.0.190
org.apache.dubbo.remoting.RemotingException: client(url: dubbo://10.20.2.19:20880/com.joyoung.wsc.product.api.service.ProductService?anyhost=true&application=wsc-gateway&check=false&codec=dubbo&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&heartbeat=60000&init=false&interface=com.joyoung.wsc.product.api.service.ProductService&loadbalance=roundrobin&metadata-type=remote&methods=saveSku,updateProduct,getById,addProduct,listProduct,listH5mProduct,deleteSku,deleteProduct,updateProductStatus,getProductById&pid=1&qos.enable=false&register.ip=172.17.0.190&release=2.7.8&remote.application=wsc-product&retries=0&revision=1.0-SNAPSHOT&service.filter=-exception&side=consumer&sticky=false&timeout=60000) failed to connect to server /10.20.2.19:20880, error message is:Connection refused: /10.20.2.19:20880
    at org.apache.dubbo.remoting.transport.netty4.NettyClient.doConnect(NettyClient.java:169)
    at org.apache.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:191)
    at org.apache.dubbo.remoting.transport.AbstractClient.reconnect(AbstractClient.java:247)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.reconnect(HeaderExchangeClient.java:166)
    at org.apache.dubbo.remoting.exchange.support.header.ReconnectTimerTask.doTask(ReconnectTimerTask.java:49)
    at org.apache.dubbo.remoting.exchange.support.header.AbstractTimerTask.run(AbstractTimerTask.java:87)
    at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:648)
    at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:727)
    at org.apache.dubbo.common.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:449)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.20.2.19:20880
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

GTaurus commented 2 years ago

dubbo服务单独分组，或者单独分名称空间

dubbo.registry.group = dubbo

or

dubbo.registry.parameters.namespace = dubbo

ljluestc commented 1 year ago

To specify the internal network segment of the container for Dubbo consumption, you can try the following methods:

In the configuration of the Dubbo consumer side, set the Dubbo registration address to the internal network segment (Pod IP address) of the container instead of the load balancing IP address. In this way, the consumer will directly access the Dubbo service inside the container.

If you are using Spring Cloud Alibaba's Nacos as the registration center, you can try to set the registration address of the Dubbo service to the internal network segment of the container in the registration configuration of Nacos.

After this configuration, the consumer of Dubbo will directly access the Dubbo service inside the container without going through the load balancing service.

apache / dubbo