[Feature][CICD]Support git action integtation test

[casionone]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Problem Description

Background Although some modules currently have unit tests, for the submitted large feature pr, manual code review cannot guarantee whether the service can run normally after the code is merged. For the pr submitted by community students, there is currently no better way to perform automated basic function verification (only the compilation can be guaranteed), and the availability of the service requires manual verification. There is an urgent need for automated integration testing to complete automated testing of new features/service availability after major revisions.

Achieved goals The service can be automatically deployed and verified after the pr is built through git action

1. Whether the service starts normally
2. Whether the basic engine tasks can be submitted normally (the verification of shell/jdbc/python/tasks is prioritized)

How to do

• Based on git action tools such as Build k8s Cluster or minikuber, use containerized deployment services for verification
• https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster
• https://github.com/marketplace/actions/minikube-single-node-kubernetes-action
• https://www.testcontainers.org/modules/docker_compose/
• testcontainers allows you to start the Docker container through a programming language, and automatically shut down the container after the program test is completed.
• https://github.com/search?q=org%3Aapache+Testcontainers&type=pullrequests

Action general flow

• Build Image to build a mirror apache/linkis:latest and use git cation cache to facilitate other actions

• minikuber load mirror apache/linkis:latest

• Inspection of kubectl commands

• Specify yaml to start the service through kubectl

• Check whether the linkis service is started normally

• Run the script to verify the underlying service

• Write integration tests for some services by using the testcontainers tool

  Tasks

• git action implements linkis code to build mirror and pull up service

• Write integration test cases based on the testcontainer component (differentiated by service)

• ---

  背景 当前虽然部分模块有单元测试，但是对于提交的大特性pr ，人工的code review无法保证合并代码后，服务是否可以正常运行。对于社区同学提交的pr ，当前没有较好的方式，进行自动化的基础功能验证（只能保证编译通过），服务的可用性需要人工验证。迫切的需要自动化的集成测试，完成对新特性/大修改后的服务可用性进行自动测试。

实现的目标 能通过git action 对pr 进行构建后自动部署服务，并验证 1.服务是否正常启动 2.基础引擎任务是否可以正常提交(优先实现 shell/jdbc/python/任务的验证)

实现大体思路

• 基于Build k8s Cluster 或则minikuber 等git action 工具 使用容器化方式部署服务进行验证
• https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster
• https://github.com/marketplace/actions/minikube-single-node-kubernetes-action
• https://www.testcontainers.org/modules/docker_compose/
• testcontainers 能够让你实现通过编程语言去启动Docker容器，并在程序测试结束后，自动关闭容器。
• https://github.com/search?q=org%3Aapache+Testcontainers&type=pullrequests

Action 大体流程

• Build Image 构建出镜像 apache/linkis:latest 并且使用git cation 缓存 方便其他动作使用
• minikuber load 镜像 apache/linkis:latest
• kubectl 命令的检查
• 通过kubectl 指定yaml 启动服务
• 检查linkis 服务是否正常启动
• 运行脚本验证基础服务
• 通过使用testcontainers 工具 针对某些服务 编写集成测试代码

主要task

• git action 实现linkis代码构建镜像和拉起服务
• 基于 testcontainer 组件 编写集成测试案例 （按服务区分）

Description

No response

Use case

No response

Solutions

No response

Anything else

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

[github-actions[bot]]

:blush: Welcome to the Apache Linkis community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

Community

WeChat Assistant	WeChat Public Account

Mailing Lists

Name	Description	Subscribe	Unsubscribe	Archive
dev@linkis.apache.org	community activity information	subscribe	unsubscribe	archive

[casionone]

task need to do Completed

• The mirror automatically builds the docker deployment, and the service is pulled up. The basic verification of the shell and the python engine is ok.

Things to optimize

• Pod service log upload (git action tool), when abnormal, you can view the service log on the git workflow page, which is convenient for locating and troubleshooting problems
• Support and verification of other major engines such as spark hive flink, etc. Based on the previous ldh mode
• Optimize the build cache and cache basic packages such as apache-hive-3.1.3-bin.tar.gz to avoid time-consuming downloads every time you build
• A new module is currently added that depends on github/actions/kind-action. Because the official whitelist for kind-action is not opened by default, it cannot be used directly. Consider submitting an official ticket to open it
• Consider introducing jmeter-type action tools such as /https://github.com/marketplace/actions/perfaction-for-jmeter to integrate testing of commonly used interfaces

• ---

  已完成

• 镜像自动构建 docker部署，和服务拉起 已经ok shell 和python 引擎基础验证 已ok

待优化事项

• pod服务日志上传（git action工具 ），异常时 在git 工作流页面能查看到服务日志 方便定位排查问题
• 其他主要引擎的支持和验证 如spark hive flink等 基于之前的ldh模式
• 优化构建缓存 ，将 apache-hive-3.1.3-bin.tar.gz 等基础包进行缓存，避免每次构建时都下载导致耗时增加
• 当前新增了模块依赖 github/actions/kind-action 因为对于kind-action 官方未默认开白名单，不能直接使用，考虑提官方工单开通
• 考虑引入jmeter 类型的action工具 如/https://github.com/marketplace/actions/perfaction-for-jmeter 集成对常用接口的测试