Xunzhuo commented 1 year ago

What would you like to be added:

Support pod injection to deploy Layotto as a sidecar in Kubernetes.

Why is this needed:

Deploying Layotto as a sidecar is a common scenario in Kubernetes.

We need to provide such things in Layotto:

Command Line Tool

A command lint tool which supports manually inject like istioctl kube-inject:

lyoctl kube-inject -f kube-resources.yaml \
--injectConfigFile inj-template.tmpl \
--layottoConfigFile layotto.json \
--valuesFile values.json

MutatingWebhook

An optional component to dynamically inject Layotto when target pod is creating.

Auto inject into pods conditions:

Namespace with label: mosn-injection: enabled
Pod without label: sidecar.mosn.io/inject: false

Please refer to injection logic in Istio.

github-actions[bot] commented 1 year ago

Hi @Xunzhuo, Thanks for opening an issue! 🎉

Xunzhuo commented 1 year ago

/kind ospp /area installation /priority high /kind hard

Xunzhuo commented 1 year ago

/milestone v0.6.0

Xunzhuo commented 1 year ago

/assign

Assign myself too, to keep an eye on it.

wenxuwan commented 1 year ago

@zhenjunMa 我们可以把这个课题加到开源之夏里面去 #894

JasonChen86899 commented 1 year ago

hi layotto member I saw layotto project recently, it is very good! I am quite familiar with knative and istio so we can add serverless feature to layotto. I hope this is a good feature.

Maybe we need an new issue for this discussion an new project like layotto-cloud-native for this.

seeflood commented 1 year ago

@JasonChen86899 Thanks! We can start from discussing the design. A design proposal is welcomed. We can use this issue or create a new issue, both are fine

Xunzhuo commented 1 year ago

Thanks @JasonChen86899, welcome to comment your ideas and proposals.

y-ykcir commented 1 year ago

@Xunzhuo Hi, I am interested in this issue and would like to have it as my OSPP 2023 program. Can you suggest where I should start？

Xunzhuo commented 1 year ago

@y-ykcir Can you send an application from OSPP website?

y-ykcir commented 1 year ago

@y-ykcir Can you send an application from OSPP website?

@Xunzhuo Sure, the submission of the application requires a proposal, and I may need some time to prepare it.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

Xunzhuo commented 1 year ago

cc @xiaoxiang10086, you can use this issue to track the status of the task.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

zhenjunMa commented 1 year ago

@Xunzhuo Hi, if the solution design has already been completed, please share it first so that we can also refer to it.

Xunzhuo commented 1 year ago

@Xunzhuo Hi, if the solution design has already been completed, please share it first so that we can also refer to it.

cc @xiaoxiang10086

xiaoxiang10086 commented 1 year ago

@zhenjunMa

This project mainly implements two modules:

A command-line tool that supports manual injection of the Layout sidecar
Kubernetes webhook component that support dynamic injection of Layotto sidecar's

I will introduce the current specific design ideas of the two modules in turn.

Command line tool

(1) layoctl kube-inject use case

layoctl kube-inject -f kube-resources.yaml \
--injectConfigFile inj-template.tmpl \
--layottoConfigFile layotto.json \
--valuesFile values.json \
--output kube-resources-injected.yaml

The kube-inject subcommand is used to inject the Layotto sidecar into the Kubernetes payload. The list of available parameters is as follows:

Option	Abbreviation	Description
--filename	-f	Kubernetes resource filename
--injectConfigFile		injection configuration file name
--layottoConfigFile		Layout configuration file name
--valuesFile		Injection values configuration file name
--output	-o	Output resource file name after injection (default value '')

(2) Implementation ideas of command line tools

Use cobra framework to parse command line arguments
Read the specified Kubernetes resource file and parse out the Pod object *corev1.PodSpec to be injected
Use Go text/template package to render injectConfigFile with valuesFile.
Use injectConfig to update Containers property value of podSpec
Write updated Kubernetes resources to the output stream

(3) Command line file

inj-template.tmpl

templates:
   sidecar: |-
     spec:
       containers:
       - name: layoutto-proxy
         image: docker.io/layotto/layotto:{{.Values.global.tag}}

values.yaml

global:
   tag: latest

layotto.json

Take the example configuration file: https://github.com/mosn/layotto/blob/HEAD/configs/config.json

(4) Questions

According to Layotto Configuration File Introduction, Layotto starts Need to read a configuration file in json format. This configuration file should already be included in the image file, so is it necessary to provide the configuration parameter layouttoConfigFile here?

Official image of Layotto found: https://hub.docker.com/r/layotto/layotto/tags

MutatingWebhook Component

(1) Background summary

Mutation Webhook is a kind of Webhook in Kubernetes, which is used to intercept and modify the mutation operation of Kubernetes objects. When the Kubernetes API Server receives a user request for an object, Mutation Webhook can intercept the request and modify the request, and then send the modified request to the Kubernetes API Server, thereby realizing dynamic modification of the Kubernetes object.

(2) Injection conditions

The Mutating Webhook component should only inject the Layotto Sidecar on pods that are in a namespace with the label mosn-injection: enabled. The pod itself has no label sidecar.mosn.io/inject: false.

(3) Implementation ideas

Creates a mutating webhook that is registered with the Kubernetes API server and is called when the Pod is created.
In MutatingWebhookConfiguration, use namespaceSelector to match namespaces with label mosn-injection: enabled.
In webhooks, sidecar injections to pods with label sidecar.mosn.io/inject: false are ignored. If the Pod meets the injection conditions, use the specified Laotto configuration file and value template file to generate a Laotto sidecarTemplate and inject it into the Pod as a sidecar.
Encapsulate the modification operation PatchOperation to return to the Kubernetes API server in AdmissionReview.Response in order to create the Pod.

Xunzhuo commented 1 year ago

@xiaoxiang10086 Great job! Hope this can land successfully : ) Ping me or @zhenjunMa if you have some questions.

Draft PRs are welcomed.

JasonChen86899 commented 1 year ago

Hi @xiaoxiang10086 @Xunzhuo a good design for injection but I have a question. In Mosn doc, it is supported by istio, maybe we can also use the same way. The job in this design do the same thing like istio inject function such as istioctl and k8s webhooks. I read layotto not much but find it dependent on mosn. Is layotto integration with mosn and only need one container? @Xunzhuo If so use istio existing function maybe another design, just like istioctl manifest apply --set .values.global.proxy.image=${LAYOTTO IMAGE} --set meshConfig.defaultConfig.binaryPath="/usr/local/bin/xxx --set meshConfig.defaultConfig.configPath="/xxx" : )

wenxuwan commented 1 year ago

Hi @xiaoxiang10086 @Xunzhuo a good design for injection but I have a question. In Mosn doc, it is supported by istio, maybe we can also use the same way. The job in this design do the same thing like istio inject function such as istioctl and k8s webhooks. I read layotto not much but find it dependent on mosn. Is layotto integration with mosn and only need one container? @Xunzhuo If so use istio existing function maybe another design, just like istioctl manifest apply --set .values.global.proxy.image=${LAYOTTO IMAGE} --set meshConfig.defaultConfig.binaryPath="/usr/local/bin/xxx --set meshConfig.defaultConfig.configPath="/xxx" : )

@JasonChen86899 Very useful information，Mosn and Layotto is one container.

zhenjunMa commented 1 year ago

According to Layotto Configuration File Introduction, Layotto starts Need to read a configuration file in json format. This configuration file should already be included in the image file, so is it necessary to provide the configuration parameter layouttoConfigFile here?

@xiaoxiang10086 Yes, the configuration file in the Layotto image is only a demo, which includes all the components supported by Layotto, most of which may not be necessary for the application. Therefore, in practical use cases, the application owner should provide this configuration file and only start the components required by the application.

wenxuwan commented 1 year ago

Dapr

dapr当前支持两种配置文件注入： ● 静态配置（--resources-path参数） ● 动态配置（通过operater）

Layotto

Layotto当前并不支持动态注入的能力，最终的形态肯定是上述的形态，Layotto适配Dapr的Operator，然后可以做组建的动态加载：

但目前无法支持，因此需要在启动的时候就注入应用的配置文件，可以injector来实现文件目录的映射：

渲染完成后类似下面的形式:

这样可能会存在一个问题，在Layotto启动完成时，如果Mount操作未完成，Layotto是无法启动完成的，可以包装个start.sh脚本，循环监听特定目录下的文件，只有mount成功后，再启动Layotto：

#!/bin/bash

while true
do
    # 判断 /data 目录下是否有名为 layotto.json 的文件
    if [ -f /data/layotto.json ]; then
    # 如果有，执行 layotto start -c layotto.json 命令
        layotto start -c /data/layotto.json
    fi
done

https://docs.dapr.io/getting-started/tutorials/configure-state-pubsub/ dapr通过静态配置和动态配置启动初始化component https://docs.dapr.io/operations/hosting/kubernetes/kubernetes-volume-mounts/ dapr的目录映射

github-actions[bot] commented 12 months ago

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

github-actions[bot] commented 11 months ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue or help wanted. Thank you for your contributions.

seeflood commented 9 months ago

这样可能会存在一个问题，在Layotto启动完成时，如果Mount操作未完成，Layotto是无法启动完成的，可以包装个start.sh脚本，循环监听特定目录下的文件，只有mount成功后，再启动Layotto：

@wenxuwan 是不是把 json 文件以config map 的形式挂载进 pod就行，这样就不用纠结启动顺序的问题了;

用 config map 来维护组件配置的话，后续可以watch 这个文件变更，有变更就通过动态配置下发机制调layotto 接口、变更组件配置这样就不用维护operator了

哦刚看了下实现，现在就是用configmap存的json

mosn / layotto

Support Pod Injection to deploy Layotto as a sidecar in Kubernetes. #910

Command Line Tool

MutatingWebhook

Command line tool

MutatingWebhook Component

Dapr

Layotto