Create CI/CD pipeline and infrastructure for meta-ros

[robwoolley]

Which CI platforms are of interest to the group?

• Jenkins, GitLab CI, Tekton, GitHub Actions, etc
• @moto-timo had previously endorsed Tekton and K8s but now advocates for simple tools for automation:
• He suggested https://laminar.ohwg.net/docs.html and https://concourse-ci.org/
• @robwoolley has used popper in the past (no longer maintained): https://github.com/getpopper/popper
• He has also used Act to run GitHub Actions locally: https://github.com/nektos/act/releases

Which workflows are of interest?

• Classic: ie. bitbake or SDKs (Containers optional)
• On-target: ie. Building with bitbake, SDK, or natively on OpenEmbedded (incl. QEMU)
• Self-hosted: ie. GitLab for a single user (gitaly, container/package registries) with sstate cache and download dir; (Containers optional)
• On-premise: ie. GitLab for multiple users using tools like Harbor and Ceph / MinIO on self-managed infrastructure with RBAC & certificates; (Containers optional)
• Cloud: ie. Using a cloud environment with cloud services for storage and orchestration (eg. Cloud Service Provider (CSP) or Cloud Development Environment (CDE))
• Others?

References

• See #1071 for issues specific to GitHub Actions

• DRM-CI: A GitLab-CI pipeline for Linux kernel testing

• Also see https://lore.kernel.org/lkml/20240228225527.1052240-1-helen.koike@collabora.com/

• https://docs.mesa3d.org/ci/index.html

• FreeDesktop / Mesa CI Gantt Charts

• ROS CI

• ROS Build Farm - Based on Jenkins, Puppet, Reprepro, GitHub, and Ubuntu

• ROS GitLab CI - Uses OSRF Docker images to compile packages for ROS 1

• ROS Industrial (industrial_ci) - Supports GitLab

• Yocto Summit Talks related to CI/CD

• Highly Scalable Build Automation by Paul Barker, Yocto Project Dev Day Virtual, North America, 2020 (Slides)

• “Gitlab CI for dummies” by Jon Mason, Bruce Ashfield, Yocto Project Summit 2021.05 (Slides, Video)

• Automating Yocto image testing with Gitlab CI/CD, Clay Timmons, Yocto Project Summit 2022.11 (Slides, Video)

• Building a Yocto CI Pipeline with KAS, GHActions and AWS by Anna-Lena Marx, Building YPS 2023.11 - 2023/11/29 (Slides, Video)

• Yocto Layer CI Build and Test with GitHub Actions, Alex Lennon (Video)

• CI/CD for Yocto Project Maintainers With Kubernetes and Tekton Pipelines, Trevor Gamblin,Tim Orling (Video)

• Yocto Continuous Integration in a Kube - Joshua Watt, Garmin (Video)

[robwoolley]

@jonmason and @rossburton have an advanced gitlab-ci.yml for Yocto in meta-arm: https://github.com/jonmason/meta-arm/blob/master/.gitlab-ci.yml

[moto-timo]

I have some older, "not-how-I-would-design-it-now" kas/gitlab-ci.yml usage on https://gitlab.com/moto-timo/yp-dev-day_virtualization (the home server that the jobs run on lost some spinning-rust back in June/July and is now offline).

Highlights there are usage of multiconfig and a bunch of different targets for specific functional reasons.

I am happy to explain how I would do it now if I was starting over again.

[moto-timo]

Also, fairly extensive usage of GitHub Actions on the various crops projects https://github.com/crops which I am also more than happy to explain/clarify.

khem and cliff have done some good work with GitHub Actions for https://github.com/Yoedistro (and you can ask khem about self-hosted GitHub Actions runners.

[moto-timo]

Pascal has had some limited but promising success with GitHub Actions for https://github.com/meta-homeassistant/meta-homeassistant which I am also experimenting with in other areas.

[moto-timo]

I can also share the design/architecture for my "Living on Master" talk, but sadly I have no access to any of the code for that project. It never made it through the Open Sourcing process. That and subsequent experience frames my current statement of "I would not use Jenkins nor LAVA for a new project".

[bchoineubility]

What are many CI/CD for ROS and Yocto !

I aware them in now.

[robwoolley]

Support for GitLab CI has been added: https://github.com/ros/meta-ros/blob/build/.gitlab-ci.yml

GitHub Actions for superflore has been added: https://github.com/ros/meta-ros/blob/master/.github/workflows/generate_recipes.yml

Work will continue, however we've hit the MVP.