An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins

[lunny]

Internal CI/CD system

I propose to implement an internal CI/CD system that is compatible with Github workflows yaml syntax, action yaml syntax, and most action plugins. And this is in fact not only a CI/CD system, but you can also choose to not check out the code for some events for example, when an issue is posted, you want to reply to them, then you don't need to check out the code. You can just reply to that issue. So the sub-system name is ~bots~actions, if you have a better name than that, please let me know.

Pros

• Migrated repositories from Github could easily reuse the actions workflow files
• Plugins could be copied from https://github.com/actions or https://gitea.com/actions to your self-hosted Gitea instances or any git service repository address
• Transparent for the CI/CD process. You can check out all workflow files and plugin codes you are using.
• Bring version management for your CI/CD scripts and plugins
• Customize your image. Github's image for ubuntu-latest is very big but Gitea now will use node:16-buster as the default image which is about 150MB and of course you could change it yourself.

Cons

• Images are maybe very big for those small devices and have to be changed when you have new dependencies.
• Many Gitea users are familiar with Drone/Woodpecker
• More feedbacks are welcome

Reference documentation

At the moment, we lack documents, but if you want to know more details about what Github Actions looks like, please visit https://docs.github.com/en/actions.

Names chosen

Different from Github's related sub-system's name, we call the plugins action or actions. One action in fact is a repository with a file named action.yml in the root, the repository is hosted in any git service server which could be gitea.com, your self-hosted Gitea instance, and even github.com. There is a default configuration but we should also support absolute git URL as a plugin repository.

Task summary

There are many things need to do

• [x] Secrets management #22142
• [x] An event listener and task scheduler #21937
• [x] A stable communication protocol between Gitea and runners https://gitea.com/gitea/actions-proto-def
• [x] A runner based on docker https://gitea.com/gitea/act_runner
• [x] A UI that will show tasks and logs #21937
• [x] Some core actions are now in https://gitea.com/actions which currently are mirrors but may be changed if there are incompatible between Github API v3 and Gitea API v1
• [x] Support absolute plugin address like uses: https://github.com/peaceiris/actions-hugo@v2 https://gitea.com/gitea/act/pulls/14
• [x] Artifact protocol compatible with Github's, some reports could be displayed in UI directly i.e. Golang standard coverage file #22738
• [x] Service support https://gitea.com/gitea/act/pulls/5
• [x] Cron support #26655
• [x] Cache action https://gitea.com/gitea/act_runner/pulls/25
• [ ] The default plugin programming language is Javascript/Typescript, but we need to support more programming languages
  • [ ] Python https://gitea.com/gitea/act_runner/issues/44
  • [x] Golang
  • [ ] Rust
  • [ ] PHP
  • [ ] More
• [ ] Check tab in the pull request to list results on the pull request page so that we don't need to go to actions page
• [ ] Composite workflow, a workflow file could reference another workflow as a step.

Although we will have an internal CI/CD system Gitea will still be open to integrating external systems and will provide more features to make the integration smooth. The below tasks do not belong to the internal CI/CD system but are very important for integrating the external CI/CD system

• [x] A standard protocol to get tasks and get logs back https://gitea.com/gitea/actions-proto-def

[a1012112796]

tips about api for ci tools to repot detail message:

1. lint messages, can show them like code comment (but can marked as wrong by manager) the stype of data that provide by api

   {
   commit: xxxxxxx
   path: "xxxxx/xxxxx",
   line: -1,
   type:  bug/ security/ codesmell/ info,
   detail: xxxxxxx,
   provider: xxxxx,
   }

example in gitee

2. code coverage data like codecov

   {
   commit: xxxxxxx
   path: "xxxxx/xxxxx",
   group: golang,    // only data in same group will be calculate together
   provider: xxxxx,
   start_line: xx,
   start_cols: xx,
   end_line: xx,
   end_cols: xx,
   matched: 0 -- not, .>1 -- yes
   }

do some calculate and show some usefull data on ui.

3. detal log in ci steps, should show them in an individual page like gh actions. then user can design a sample ci tools which not need it's own ui :) style:

   {
   commit: xxxxxxx
   step_name: aaa,
   parent_step_name: bbb,
   previous_step_name: ccc,
   log: xxxxx, or link to log data
   status: pending or runing or success ....
   }

   example on gh

[Be-ing]

Support github actions files and allow external agent to execute the action via Gitea's API and returned result as above results.

Can you clarify what this will entail? Would this mean GitHub Actions could run on a github.com mirror of a repo and the results would appear on a Gitea server?

[lunny]

That means Github actions could be running on Gitea(with this CI feature) servers(actually on user's agent, Gitea will only a coordinator and result saver). Then mirrors/migrations of repositories from github/github enterprise could run the ci directly and no(or small) modification needed.

[a1012112796]

see

• https://docs.github.com/en/rest/reference/checks
• https://docs.github.com/en/rest/reference/code-scanning

[moqmar]

Would this be implemented by explicitly checking for a .github directory and sending those repos to the agent automatically on push using something like a global webhook? Because I think it would be great to not limit this to just GitHub Actions, but rather have a "global webhook" for pushes to an agent that can then decide what to do with that push - either using the GitHub Actions definitions, or for example running other CI implementations.

[lunny]

@moqmar Global webhooks have been implemented. The issue is to be compatible with actions for those migrations from github.

[ChristopherHX]

I have been working on an external webhook endpoint for github actions self-hosted runners to use with my gitea instance or locally on any pc. This is basically a fork of the runner to be able to reuse their code for a missing aspnetcore server. During my tests I was able to use gitea almost like a ghes instance, some actions like actions/checkout@v2 have had minor incompatibilities with authentication.

I would be glad to see endpoints for CI logs and have an integrated UI. Currently I have a basic react UI to display my logs, added as an external tab to gitea.

Experimental Usage Download and extract https://github.com/ChristopherHX/runner.server/releases/latest for your system

Setup Gitea as an ghes endpoint, file bin/appsettings.json, change the url from http://localhost:5000 to a different http url / port

  "Runner.Server": {
    "GitServerUrl": "https://gitea:3042",
    "GitApiServerUrl": "https://gitea:3042/api/v1",
    "GitGraphQlServerUrl": null,
    "GITHUB_TOKEN": "Add your personal access token here",
    "ActionDownloadUrls": [
      {
        "TarballUrl": "https://gitea:3042/{0}/archive/{1}.tar.gz",
        "ZipballUrl": "https://gitea:3042/{0}/archive/{1}.zip"
      }
    ]
  }

Add <url of Runner.Server>/runner/server/_apis/v1/Message (http://localhost:5000/runner/server/_apis/v1/Message) as a webhook (select gitea) for your repo, to enable actions.

Setup https and use openid connect with gitea (Optional for Security, only working if the github actions server is using https)

• Create an oauth2 app (User settings)
• You will need to add the clientid and secret into bin/appsettings.json
• You will need a pem certificate pair (cert.pem (only a single certificate will work, no cert chain) , key.pem)

  "Kestrel": {
  "Endpoints": {
    "HttpsFromPem": {
      "Url": "https://*:5001",
      "Certificate": {
        "Path": "./cert.pem",
        "KeyPath": "./key.pem"
      }
    }
  }
  },
  "ClientId": "ClientId of your Oauth app",
  "ClientSecret": "Client secret of your Oauth app",
  "Authority": "https://gitea:3042",

Add <url of Runner.Server>/signin-oidc (https://localhost:5001/signin-oidc) as redirect url for the OAuth app in gitea.

Start bin/Runner.Server(.exe) Open <url of Runner.Server> (http://localhost:5000, https://localhost:5001) to see if a job was received

Configure self-hosted runners, https://github.com/ChristopherHX/runner.server#advanced-usage

EDIT 2021/10/26 I added a ( temporary ) public test instance of runner.server connected to try.gitea.io https://try.gitea.io/ChristopherHX/actions-on-gitea/commit/7ac850a3a96a7c60417c1306f50d49ea08d2e7d7

Press rerun workflow to see in progress job runs EDIT 2021/02/02 Fix openid sample config to actually have working https with pem

[techknowlogick]

@ChristopherHX whoa! That's amazing!!

[Bobby99as]

This seems nice

[silverwind]

Regarding the name: I think "bot(s)" is too confusing with "robot(s)" which are automated user accounts. How about naming it "step(s)"? Most of the terminology that relates to "action(s)" would also apply to "step(s)".

[lunny]

Regarding the name: I think "bot(s)" is too confusing with "robot(s)" which are automated user accounts. How about naming it "step(s)"? Most of the terminology that relates to "action(s)" would also apply to "step(s)".

To diff from Github Actions, I don't think we should use actions as name. Bots might means automatical but not robot, I'm not English native speaker and maybe there is a better name for it. This is really not only a CI/CD system, you can do more like below yml file.

name: Issue post

on:
  issue_comment:
    types: [created]

jobs:
  comment_created:
    runs-on: ubuntu-latest
    steps:
      - run: env
      - name: Create comment
        uses: peter-evans/create-or-update-comment@v2
        with:
          issue-number: ${{github.event.issue.number}}
          body: |
            This is a multi-line test comment
            - With GitHub **Markdown** :sparkles:
            - Created by [create-or-update-comment][1]

            [1]: https://gitea.com/peter-evans/create-or-update-comment
          reactions: '+1'

But I think maybe step(s) is too small for this feature?

[eeyrjmr]

how about automate

[silverwind]

I'm not English native speaker and maybe there is a better name for it

It's well established even in non-english languages that bot is a program that impersonates a human.

So we are looking for a noun that signifies something being done. "Build" does qualify but is already occupied in the sense of it meaning to construct software. Some ideas maybe here.

[JakobDev]

I personal think developing this makes no sense. As said in the start post, there are already working CI systems. It would make more sense to integrate e,.g. Woodpecker directly in Gitea instead of wasting a lot of development resources to develop something that already exists.

[silverwind]

How about Task(s) or Job(s)? E.g. signifying a unit of work.

[lunny]

I personal think developing this makes no sense. As said in the start post, there are already working CI systems. It would make more sense to integrate e,.g. Woodpecker directly in Gitea instead of wasting a lot of development resources to develop something that already exists.

They are different systems. Gitea could support both.

[6543]

well before merge we should make sure https://github.com/woodpecker-ci/woodpecker/issues/1385 is ready and gitea do support both equally

[6543]

As I pointed out, I don't like the feature-creep but as long as it is possible to disable it completely, I won't block this but also wont focus on fix it either ...

[MarkusAmshove]

What about Auteamate? 😋 Contains tea, a friendly mate and suggests automation

[kmk3]

Suggestions of boring and obvious names:

• Gitea Automation (looks relatively similar to "GitHub Actions")
• Gitea CI (like GitLab CI and countless others)

[kmk3]

Suggestions of tea-related names:

• Boba CI (as in boba tea)
• Bubble CI
• Gitea Brew
• Gitea Chai Infusion (Gitea CI)
• Gitea Continuous Infusion (Gitea CI)
• Teabot
• Teamagic

[williamdes]

Suggestions of boring and obvious names:

• Gitea Automation (looks relatively similar to "GitHub Actions")

I would put this one first because the name is easy and stays in mind

• Gitea CI (like GitLab CI and countless others)

Boring but just perfect for what it is 😄

[tobiasBora]

Can't we stick to a vocabulary as close as possible to github's when possible? This would allow a faster transition from people coming from github, and ideally easy migrations. Of course, if they do incompatible things we should get a new name. I prefer the "boring" but descriptive terms like "task" compared to fancy unclear terms.

I agree that it would be awesome to have native gitea integration. But if it takes ressources, I agree it would be good to disable it to run it on low-end devices (I still want gitea to run on my raspberry pi).

[lunny]

How about motions which is close to actions and short enough.

Or autos ?

[ghost]

I just tried it and wrote a note:

https://gist.github.com/xin-u/f459c4805b348595e4981000b75443fe

[lunny]

feature-creep

I don't think it's a feature-creep. The issue has been opened for 2 years and I have announced I'm working on that in discord for many times. And I think a PR should be created when it's basically ready but not from the day 1.

And I think Gitea should create a standard protocol and other external CI/CD tools could implement that to reuse Gitea's UI. Not only woodpecker, but also drone/jenkins and others could also implement the protocols to upload logs and etc. But I think that could be a standalone PR after a solid proposal.

[lafriks]

Yes would be nice if ui had some kind of abstraction to allow implement other runners inside gitea (something ui wise I started once but had no time to finish)

https://github.com/go-gitea/gitea/compare/main...lafriks-fork:gitea:feat/gitea_builds?expand=1

This way you would not have to enable it specifically, you would have to just add global/org/user/repo runners and use them

[lunny]

Yes would be nice if ui had some kind of abstraction to allow implement other runners inside gitea (something ui wise I started once but had no time to finish)

https://github.com/go-gitea/gitea/compare/main...lafriks-fork:gitea:feat/gitea_builds?expand=1

This way you would not have to enable it specifically, you would have to just add global/org/user/repo runners and use them

Looks like in your design, every runner type has his protocol and Gitea will implement different protocol for different runners?

[lunny]

For the name, I think we can have a vote.

We have several choices here(sorry this is not the whole suggested names because I think we haven't so many emojis :))

• 🚀 Actions
• 👀 Automate
• ❤️ Motions
• 👍 Bots
• 🎉 Auteamate
• 😄 Builds

continue in https://github.com/go-gitea/gitea/issues/13539#issuecomment-1332985355

[lafriks]

Yes would be nice if ui had some kind of abstraction to allow implement other runners inside gitea (something ui wise I started once but had no time to finish) https://github.com/go-gitea/gitea/compare/main...lafriks-fork:gitea:feat/gitea_builds?expand=1 This way you would not have to enable it specifically, you would have to just add global/org/user/repo runners and use them

Looks like in your design, every runner type has his protocol and Gitea will implement different protocol for different runners?

yes but with common interface that needs to be implemented (like we have for webhooks or packages)

[BySempron]

Is there only going to be one docker runner? Docker isn't the best in all setups, so I think we must have a "bare metal" runner which works without docker.

Just like AppVeyor default runner. Simply execute the commands you specify in your action file.

[ChristopherHX]

@BySempron The fork of gitea I saw didn't include my PR for a native execution environment https://github.com/nektos/act/pull/1293. I hope that this change will be pulled and made available.

[tobiasBora]

If I can give maybe another application of bare runners proposed by @BySempron, it would be for Nix (actually a gitea-native Nix runner would be also really cool… and maybe even simpler to implement as compiling in nix is just a matter of providing build instructions in some flake.nix file at the root of the repository and typing nix build path-of-repo#name-program to compile… you can also add tests and more). This makes sense when compiling softwares "packaged" using Nix as Nix already provides a sandbox restricting the scope of the building process (for security purpose and purity). And it also gives for free some sort of cache sharing among builds (like if my first software packages a package A, and my second software uses A, when the CI builds A, it will automatically be available when compiling software B without any need of recompiling software A). Also, it shares the dependencies between completely unrelated repository automatically (like if two repositories build some LaTeX documentation using the same LaTeX version, LaTeX will be downloaded just once and automatically shared among builders).

[eeyrjmr]

fyi "Actions" already refers to a Gitea UI and thus to name the worker scripts "actions" can raise some ambiguity if someone refers to them.

[TheBrokenRail]

Name-wise, what about Teapots, Pots, or Kettles? (You know, because teapot/tea kettle?)

I also like "Sips" which was suggested on the Discord.

[lunny]

@BySempron The fork of gitea I saw didn't include my PR for a native execution environment nektos/act#1293. I hope that this change will be pulled and made available.

I think we can pull upstream code again in https://gitea.com/gitea/act and maybe you can help to send more PRs to these repositories.

[lunny]

Pots

Ah, right. I think we have to change the menu name if Actions win.

[lunny]

• 👍 Sips
• ❤️ Pots
• 🚀 Kettles

[wolfogre]

The fork of gitea I saw didn't include my PR for a native execution environment nektos/act#1293. I hope that this change will be pulled and made available.

@ChristopherHX It's amazing! The forked act is trying to track the latest release of nektos/act, so I will follow up once a new version including the commit is released.

[kmk3]

@lunny Suggestion to add to the above list:

• :tada: Brews

(It works both as a noun and as a verb, so you could have a [Brew] button)

[wolfogre]

@lunny Suggestion to add to the above list:

• 🎉 Brews

(It works both as a noun and as a verb, so you could have a [Brew] button)

@kmk3 Excuse me, don't do that, please! 😂 I'm a user of homebrew, and typing brew install ... brew upgrade ... everyday.

[kmk3]

@kmk3 Excuse me, don't do that, please! joy I'm a user of homebrew, and typing brew install ... brew upgrade ... everyday.

Ah yes, maybe the CLI usage could be different then, like tea brew, gtb or something (in which case the binary name for "Gitea Actions" could be rather amusing).

Though would there even be a dedicated binary for this?

[lafriks]

I think naming should be kept in UI as generic as possible to be able to integrate under that other CI information (Woodpecker CI, Jenkins etc) under it so imho Workflows would be generic enough not to hint to any exact product but be recognizable enough. CI/CD tool that is being developed as one of integration can be named than in some more specific name.

[wolfogre]

I believe Actions is the final result. I know it's not perfect, but it's probably good enough for now.

[gnat]

Just rename the existing button to Manage (or whatever) and use Actions for this.

[ChristopherHX]

It's amazing! The forked act is trying to track the latest release of nektos/act, so I will follow up once a new version including the commit is released.

@wolfogre Thank you for syncing nektos/act, however releases are currently just a monthly snapshot of master

You can now use this platform config self-hosted:docker://-self-hosted to get a native execution environment. The name docker:// is now confusing in this specfic case, because this specfic platform tag is not using docker at all.

[gnat]

Wanted to say, as someone who uses a ton of Drone, it's going to be amazing to migrate to this new system and standardize on the Github actions format instead. Really looking forward to this.

[ibotty]

As mentioned before, I'd really like to see different runners than only ones based on docker. There is a reason that the whole red hat influenced world moved away from docker.

But more importantly, I'd like to have a runner that runs the containers in a Kubernetes cluster. Will it be possible to use a different runner or is there tight integration? Is there some preliminary documentation besides the source on the api contract between gitea and the actions runner?

[lunny]

As mentioned before, I'd really like to see different runners than only ones based on docker. There is a reason that the whole red hat influenced world moved away from docker.

But more importantly, I'd like to have a runner that runs the containers in a Kubernetes cluster. Will it be possible to use a different runner or is there tight integration? Is there some preliminary documentation besides the source on the api contract between gitea and the actions runner?

The protocol repository is https://gitea.com/gitea/actions-proto-def. I think we can have more runners in future, but let's have one at first. A runner based on act should be the easiest way to make one.

[sschuberth]

Cache action

I guess support for that would require to address https://github.com/nektos/act/issues/989#issuecomment-1156768547, which would be a great step towards running actions that leverage caches locally.