An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins #13539

Closed lunny closed 1 year ago

Internal CI/CD system

I propose to implement an internal CI/CD system that is compatible with Github workflows yaml syntax, action yaml syntax, and most action plugins. And this is in fact not only a CI/CD system, but you can also choose to not check out the code for some events for example, when an issue is posted, you want to reply to them, then you don't need to check out the code. You can just reply to that issue. So the sub-system name is ~bots~actions, if you have a better name than that, please let me know.

Pros

Migrated repositories from Github could easily reuse the actions workflow files
Plugins could be copied from https://github.com/actions or https://gitea.com/actions to your self-hosted Gitea instances or any git service repository address
Transparent for the CI/CD process. You can check out all workflow files and plugin codes you are using.
Bring version management for your CI/CD scripts and plugins
Customize your image. Github's image for ubuntu-latest is very big but Gitea now will use node:16-buster as the default image which is about 150MB and of course you could change it yourself.

Cons

Images are maybe very big for those small devices and have to be changed when you have new dependencies.
Many Gitea users are familiar with Drone/Woodpecker
More feedbacks are welcome

Reference documentation

At the moment, we lack documents, but if you want to know more details about what Github Actions looks like, please visit https://docs.github.com/en/actions.

Names chosen

Different from Github's related sub-system's name, we call the plugins action or actions. One action in fact is a repository with a file named action.yml in the root, the repository is hosted in any git service server which could be gitea.com, your self-hosted Gitea instance, and even github.com. There is a default configuration but we should also support absolute git URL as a plugin repository.

Task summary

There are many things need to do

[x] Secrets management #22142
[x] An event listener and task scheduler #21937
[x] A stable communication protocol between Gitea and runners https://gitea.com/gitea/actions-proto-def
[x] A runner based on docker https://gitea.com/gitea/act_runner
[x] A UI that will show tasks and logs #21937
[x] Some core actions are now in https://gitea.com/actions which currently are mirrors but may be changed if there are incompatible between Github API v3 and Gitea API v1
[x] Support absolute plugin address like uses: https://github.com/peaceiris/actions-hugo@v2 https://gitea.com/gitea/act/pulls/14
[x] Artifact protocol compatible with Github's, some reports could be displayed in UI directly i.e. Golang standard coverage file #22738
[x] Service support https://gitea.com/gitea/act/pulls/5
[x] Cron support #26655
[x] Cache action https://gitea.com/gitea/act_runner/pulls/25
[ ] The default plugin programming language is Javascript/Typescript, but we need to support more programming languages
- [ ] Python https://gitea.com/gitea/act_runner/issues/44
- [x] Golang
- [ ] Rust
- [ ] PHP
- [ ] More
[ ] Check tab in the pull request to list results on the pull request page so that we don't need to go to actions page
[ ] Composite workflow, a workflow file could reference another workflow as a step.

Although we will have an internal CI/CD system Gitea will still be open to integrating external systems and will provide more features to make the integration smooth. The below tasks do not belong to the internal CI/CD system but are very important for integrating the external CI/CD system

[x] A standard protocol to get tasks and get logs back https://gitea.com/gitea/actions-proto-def

I made a POC gitea actions runner proxy based on actions/runner, this doesn't use nektos/act for action execution and doesn't require docker. https://gitea.com/ChristopherHX/actions_runner

Now two runner implementations for gitea actions.

Things, which act doesn't support, but are possible with this proxy runner

service container
timeout-minutes

Bugs caused by the runnerv1 protocol, but are ok since it's experimental

actions/runner's live logs are skipping lines, but it is not possible to upload the full log without breaking live logs after the steps are finished
job outputs cannot be sent back https://gitea.com/gitea/actions-proto-def/issues/4
Not possible to update display name of steps from runner Pre and Post steps are part of setup and complete job, steps not in the job are ignored by the server

Great Job, I feel most of the people using Gitea over Gitlab and Github enterprise use it because it does not need Linux/WSL/docker stuff. Then I think actions runner being native is crucial for real cross platform support.

Great Job, I feel most of the people using Gitea over Gitlab and Github enterprise use it because it does not need Linux/WSL/docker stuff. Then I think actions runner being native is crucial for real cross platform support.

I don't have the numbers, but why would anyone run Gitea on Windows instead of a server OS like Linux? I know it can run on Windows, but especially if you allow outside access you don't want a Windows machine connected to the "evil" internet.

Then again, I can understand the need for native runners for those who haven't containerized their services yet.

I don't have the numbers, but why would anyone run Gitea on Windows instead of a server OS like Linux? I know it can run on Windows, but especially if you allow outside access you don't want a Windows machine connected to the "evil" internet.

There's software you can only build on windows, like tauri (at least for now). I have worked on a project where you could only deploy the software by pushing it to a server in the client's internal network and the vpn they were using was only available on Windows. With a Windows runner it's possible to build pipelines for these use cases which otherwise would be not possible.

Ah, Windows native runners and not "Gitea on Windows"; didn't think about that ;)

Carry on!

but why would anyone run Gitea on Windows instead of a server OS like Linux?

Guess you've never heard about Windows Server then and probably also don't know how much easier it is to find administrators who know what they're doing.

Great Job, I feel most of the people using Gitea over Gitlab and Github enterprise use it because it does not need Linux/WSL/docker stuff. Then I think actions runner being native is crucial for real cross platform support.

I don't have the numbers, but why would anyone run Gitea on Windows instead of a server OS like Linux? I know it can run on Windows, but especially if you allow outside access you don't want a Windows machine connected to the "evil" internet.

Then again, I can understand the need for native runners for those who haven't containerized their services yet.

Linux servers are more popular over Windows server on the internet but there are many Enterprise Environments that reply on Active Directory stuff and and use Windows Servers, By the way my purpose was that among Gitea/Github/Gitlab, The Gitea is the only one support Deploying to Windows Servers. so all individuals or companies who stick with windows have no option but Gitea.

Regarding absolute plugin address, please consider requiring to write a http:// / https:// prefix for absolute addess plugins.

Support absolute plugin address like uses: github.com/peaceiris/actions-hugo@v2

In my opinion is github.com/peaceiris/actions-hugo@v2 ambigious due to the following reasons:

For example the cache action (https://github.com/actions/cache/blob/main/restore/README.md) has child plugins in subfolders, which nektos/act does support. uses: actions/cache/restore@v3 is valid in github action, you cannot count slashes to resolve this parsing problem
Imagine you create a user or org called github.com, gitea (1.17.2, I'm too lazy to apply updates) doesn't forbid to use dot in names. What will happen if the DEFAULT_ACTION_URL is that gitea instance? I couldn't tell the difference of user github.com and a dns name.

I have no concerns regarding the following notation: uses: https://github.com/peaceiris/actions-hugo@v2, it's very uncommon to repeat a / in a uses statement and easy to parse.

@ChristopherHX those are very good points. To confirm, your suggestion is to treat anything that doesn't start with http/s as relative plugin to the default action url, but if it starts with http/s then use absolute? As I think that is a very reasonable suggestion.

your suggestion is to treat anything that doesn't start with http/s as relative plugin to the default action url, but if it starts with http/s then use absolute?

Yes exactly.

Will Gitea CI support buildpacks? https://buildpacks.io/

Cloud Native Buildpacks transform your application source code into images that can run on any cloud.

Why was not used the protocol between gitlab and there runner? this runner could be runned everywhere and start jobs in virtualbox, shell, kubernetes, docker ....

Why was not used the protocol between gitlab and there runner? this runner could be runned everywhere and start jobs in virtualbox, shell, kubernetes, docker ....

GitLab runners are designed for .gitlab-ci.yml, however, what Gitea wants is Actions.

But for Actions does not exists an good runner, with service, running in kubernetes and so on. Why has gitea the same API then github?

Why was not used the protocol between gitlab and there runner? this runner could be runned everywhere and start jobs in virtualbox, shell, kubernetes, docker ....

I think we can discuss the possibility of whether we can support that protocol and GitLab runners.

An overview how Gitea Actions are different compared to GitHub Actions, bugs are included.

From act side

concurrency is ignored
run-name is ignored
context availability is not checked, so you can use the env context on more places
~~service container not implemented~~ Available in latest builds of act_runner (Update 19. April 2023, no release yet)
step summary not processed
problem matcher ignored
annotations ignored
no vars context (Work in Progress) (Update 19. April 2023)
incomplete github context
run steps on cancellation not implemented
depending if act_runner runs as a service a tty is present or not for container jobs, always a tty if supported on native runner (GitHub has never a tty available)
job.permissions block ignored
timeout-minutes ignored
job.continue-on-error ignored (idk what gitea is doing)
PATH of container / of act must contain node for nodejs actions, github runner has their own copy for both container and host
Openid Connect url is not defined
job.environment ignored and scoping secrets by deployment environment is not supported

No support for new runs-on syntax

runs-on: # Setting this to `{}` doesn't work reliable on github.com, an self-hosted runner may pick this up regardless of assigned labels or runner group
labels: a, b # Optional
group: somegroup # Optional

From gitea side

Job conditions are evaluated on the runner not on gitea
needs context not available (~~job results are not yet part of the protocol~~) (Work in Progress) (Update 19. April 2023)
only static matrix strategy, cannot use json from previous job output. I'm not shure if the expressions are evaluated, confirmed that only runs-on is evaluated on gitea side as of now
~~no reusable workflows (on act only partially implemented, not checked)~~ partial reusable workflow support on the act_runner side (needs latest builds of gitea to pass with and secrets), GitHub does it on their server (Update 19. April 2023)
max job time for container jobs has a hardcoded limit below 6hours, native runner has no timeout
job cancellation request not sent to runner
step.ContinueOnError cannot be an expression, model is a copy in jobparser pkg of act fork
Non functional differences
pre and post steps don't have their own section in the job log ui
workflow (syntax) errors are not reported in the webui
Things gitea have github don't
absolute actions via http(s):// protocol, similar to docker://
go actions (GitHub requires to use composite actions to do the same)

This list has been originally posted in the gitea discord channel EDIT last update 19. April 2023

Encouraging for the "standardising on Actions" story for Gitea... from Django Developers Survey 2022.

I'm already converting many of my projects over for the Gitea transition to Actions.

but you still need build it yourself to test it?

For a mirror repository, Will it be possible to use different action instead of the one in the original repository?

For example, lets say the github action in https://github.com/go-gitea/gitea compiles gitea when a new tag is created with TAGS="bindata sqlite sqlite_unlock_notify" make build.

I set up a mirror repository at https://git.mydomain.com/me/gitea and in my gitea action, I would like to it to compile without sqlite support, TAGS="bindata" make build.

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

I think it would be nice to be able to select which folder to use for Gitea Actions via a global and project level flag, with the project level flag taking priority.

That way Gitea Actions can ignore the .github folder in favor of the .gitea folder.

You can't modify files in mirrored repository.

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

It will be better to have a event hook that will trigger actions in another repository, so mirror would trigger a hook, that hook triggers an actions run in another repository which will allow you to specify whatever you want.

Gitea 1.19.0 was released with the initial actions implementation, but I do not see any mention of it at https://docs.gitea.io/. So maybe a new task should be added to the checklist in the issue description?

Document Gitea actions setup and configuration

It is also well noted we will also need to add Helm chart for act

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

Being able to store the configuration in a custom branch would help a lot for adding automation to open source projects that don't want to have specific CI files added to the core repo. The linux kernel for instance.

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

Being able to store the configuration in a custom branch would help a lot for adding automation to open source projects that don't want to have specific CI files added to the core repo. The linux kernel for instance.

Maybe we can have some options: 1) .github/workflows or .gitea/workflows 2) .actions.git like .wiki.git 3) A setting of the repository to store multiple workflow files. 4) A global workflows feature, which you can create a workflow to includes files and repositories.

I actually really like the idea of having a seperate repo analogous to the wiki repository for this. Would be interesting to explore other functionality that could be implemented in this way.

Yup now I've been exploring more, I can see a separated repo for the workflows could be a very helpful way of doing this too. Then multiple repos (mirrors, forks, or separate repos for separate developers) can point to common workflow actions.

Maybe after artifact upload is done, basic code intelligence (not as powerful as SourceGraph, but just goto definitions/references + browse file symbols) could be implemented. After finishing builds/tests the action might do something like rust-analyzer lsif . and upload it as a special artifact Gitea will build navigation from. This approach gives the most detailed code navigation, just as in IDE using the same language server as used to generate LSIF. See https://lsif.dev/ for more details regarding the format.

Didn't open a new issue because of dependency on artifacts

Are there any docs on built-in actions in 1.19.0? Even just a quick bullet point here in this thread? 🙏

Answering my own question, getting this up and running with Gitea v1.19.0 (from upstream Docker images) was pretty straight forward:

Add the following to your app.ini config (since if you're running an older version, it won't be there by default and disable by default?):

[actions]
ENABLED = true

Restart Gitea
Go to Site Administration -> Runners
Create new Runner and Copy the token
Setup and register a Runner

$ git clone https://gitea.com/gitea/act_runner.git
$ cd act_runner
$ make build
$ ./act_runner register
# follow the instructions
$ ./act_runner daemon

That's it! 🥳 Now just add workflows in .gitea/workflows/build.yml which use the same syntax as Github actions? (I guess so since the runner being used is the act_runner).

Happy to continue beta testing this (I also run an older Drone CI setup too!) and provide feedback.

One thing that springs to mind (which I'm hoping one of the developers can answer):

How do I set-up Gitea + actions in a way where I can spawn runners on demand? (like Github does)

Basically, instead of having a static pool of runners (however I choose to run them), it would be nice to have a webhook (globally) that I can listen on and spawn a new runner on demand (say in a virtual machine or similar). Thanks! 🙏

Apparently in version 19.0.1 we should put something different in the app.ini file. Can someone help with that? I put the below in it and actions weren't enabled. I used docker image by the way... I don't know if that's the source of issue. Really poor documentation so far...

[repo.actions]
ENABLED = true

@aminify I think it's just:

[actions]
ENABLED = true

https://docs.gitea.io/en-us/config-cheat-sheet/#actions-actions

No, the cheatsheet is still true. You simply misunderstood something here: the key DEFAULT_REPO_UNITS/DEFAULT_FORK_REPO_UNITS in the [repository] section used the value actions instead of repo.actions. The section to enable actions is [actions].

All units configuration should use repo.actions, but the ini section is [actions].

How do I set-up Gitea + actions in a way where I can spawn runners on demand? (like Github does)

This would be nice! I've been tinkering with using it to deploy/restart my docker services. Would be nice if it could just do dynamic generation / access from outside for runner tokens. That way the container is more ephemeral.

Also its probably been said before, but an environment variable for enabling actions would be nice, instead of having to update app.ini.

I have setup the runner and everything and I have a docker build command to run in my action. Unfortunately I get /var/run/act/workflow/1: line 2: docker: command not found. Anyone knows the best way to enable docker inside the runner?

I have setup the runner and everything and I have a docker build command to run in my action. Unfortunately I get /var/run/act/workflow/1: line 2: docker: command not found. Anyone knows the best way to enable docker inside the runner?

You need to install it or use a docker image runner that has it installed

thanks @panekj for others facing the same issue. I managed to fix this docker: command not found issue using ubuntu-latest:docker://catthehacker/ubuntu:act-22.04 as label argument for act-runner (in register phase)

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

Wait?! Why are you logging secrets in the first place? 🤔

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

I can confirm this. I use the 1password action to retrieve passwords. They are logged in all their beauty. Their code says add-mask though.

Wait?! Why are you logging secrets in the first place?

There are Programs who print API Keys for debug reasons with no Way to disable it. It also possible, that this may be leaked in a error message when something crashes, so it's generally a good Idea to mask this, just in case.

Yup now I've been exploring more, I can see a separated repo for the workflows could be a very helpful way of doing this too. Then multiple repos (mirrors, forks, or separate repos for separate developers) can point to common workflow actions.

FWIW - Today I discovered gitlab can do this and it's all documented here:

Wait?! Why are you logging secrets in the first place?

There are Programs who print API Keys for debug reasons with no Way to disable it. It also possible, that this may be leaked in a error message when something crashes, so it's generally a good Idea to mask this, just in case.

Those programs should be loaded up on a rocket and shot into sun! 😆 talk about security law practice 🤦‍♂️

I wonder how much more we want to abuse this one issue? As far as I'm concerned the basic functionality exists, and the remains should be extracted into separate issues. I don't think it's good that this issue is still open despite the main functionality being implemented already. Issues should focus on one specific thing, otherwise they'll be pretty much never closed.

go-gitea / gitea