search

alibaba / open-local

cloud-native local storage management system for stateful workload, low-latency with simplicity
Apache License 2.0
463 stars 81 forks source link

Ftr: add option of use node hostname dns to support edge scene #207

Closed LaurenceLiZhixin closed 1 year ago

LaurenceLiZhixin commented 1 year ago

In the edge computing scenario, we need to go through the cloud edge network proxy to access the specific port of the edge computing node, so I added a start option, use-node-hostname-dns, which defaults set to false, to enable the grpc call based on the hostname.

After this parameter is enabled, the machine will no longer based on an ip address to dial. Instead, the request for with dial to ${hostName}:${port} will be made, for example, my-edge-001-node:1736. The domain name resolves the dns server configured during deployment to the proxy, which is applicable to the edge computing scenario where the network is visible in one direction.

在边缘计算场景下，我们需要通过云边网络代理，才能访问至边缘计算节点的特定端口，因此我增加了一个启动参数 use-node-hostname-dns， 默认为 false，用于开启基于主机名的grpc 调用。

启动这一参数后，位于中心侧的 csi-plugin 在建立grpc 链接之前，将不再基于机器ip进行调用，而是直接发起 ${hostName}:${port} 的请求，例如 my-edge-001-node:1736 。域名将被部署至集群时时配置的 dns 服务器解析至代理，从而适配于网络单向可见的边缘计算场景。

我们的架构如下：

image

CLAassistant commented 1 year ago

CLA assistant check
All committers have signed the CLA.

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 30.00% and project coverage change: +0.25 :tada:

Comparison is base (c01fc51) 32.07% compared to head (b122fac) 32.33%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #207 +/- ## ========================================== + Coverage 32.07% 32.33% +0.25% ========================================== Files 41 41 Lines 6344 6352 +8 ========================================== + Hits 2035 2054 +19 + Misses 4022 4009 -13 - Partials 287 289 +2 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `32.33% <30.00%> (+0.25%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/alibaba/open-local/pull/207?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba) | Coverage Δ | | |---|---|---| | [pkg/csi/driver.go](https://codecov.io/gh/alibaba/open-local/pull/207?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba#diff-cGtnL2NzaS9kcml2ZXIuZ28=) | `0.00% <0.00%> (ø)` | | | [pkg/csi/controllerserver.go](https://codecov.io/gh/alibaba/open-local/pull/207?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba#diff-cGtnL2NzaS9jb250cm9sbGVyc2VydmVyLmdv) | `58.66% <50.00%> (-0.19%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://codecov.io/gh/alibaba/open-local/pull/207/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

peter-wangxu commented 1 year ago

两个问题： @LaurenceLiZhixin

  1. 这个场景是不是csi controller链接节点 lvmd的优先地址类型？跟apiserver链接kubelet有些类似

    --kubelet-preferred-address-types strings     默认值：Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP 用于 kubelet 连接的首选 NodeAddressTypes 列表。 如果是的话，要扩展性好的话可以参考apiserver类似的实现（是不是跟你们边缘场景更加匹配？）

  2. 这个配置看上去是全局一致的，所有边缘节点都使用同样的逻辑链接，是否存在不同的节点使用不同的地址类型？

LaurenceLiZhixin commented 1 year ago

NodeAddressTypes @peter-wangxu

  1. 第一个问题，你说的对，是这样，但是鉴于现在 openlocal 对于这个参数可扩展性的需求有限，我觉的暂不必要参考apiserver 的实现。
  2. 第二个问题，开启这一开关的 csi-plugin ，在我们的场景下是位于中心测的；位于边缘节点的实例无需开启。
TheBeatles1994 commented 1 year ago

NodeAddressTypes @peter-wangxu

  1. 第二个问题，开启这一开关的 csi-plugin ，在我们的场景下是位于中心测的；位于边缘节点的实例无需开启。

Agent是以daemonset方式部署，一个集群同一组件应该只支持一种配置吧？

LaurenceLiZhixin commented 1 year ago

NodeAddressTypes @peter-wangxu

  1. 第二个问题，开启这一开关的 csi-plugin ，在我们的场景下是位于中心测的；位于边缘节点的实例无需开启。

Agent是以daemonset方式部署，一个集群同一组件应该只支持一种配置吧？

是的，我只是在中心侧用到这个参数，边缘侧不做改动。

TheBeatles1994 commented 1 year ago

2. 第二个问题，开启这一开关的 csi-plugin ，在我们的场景下是位于中心测的；位于边缘节点的实例无需开启。

openyurt 支持这种同一集群下同一 daemonset 在边缘节点的 pod 配置与中心侧不同？

LaurenceLiZhixin commented 1 year ago

在 CNStack 的架构中，中心侧的 controller 是 deployment 部署的，并不是ds、我只需要针对deploy yoda-controller -c csi-plugin 这一容器开启参数即可 @TheBeatles1994

peter-wangxu commented 1 year ago

LGTM

TheBeatles1994 commented 1 year ago

备注：边缘场景下，controller调用同一 pod 内的 csi-plugin 容器，该容器使用 node name 去连接对应节点（边缘节点、非边缘节点均可）的 agent。这里边缘组件会提前配置集群的 dns，同时 controller 中的 csi-plugin 容器也会配置 dns 配置。