Migrate to GitHub Actions

[PeterBowman]

Travis CI is now making $$$ on "free" builds according to consumed job minutes (ref). Active users/orgs will hit the free plan limit sooner or later. It had happened to YARP folks and they decided to switch to GH Actions at https://github.com/robotology/yarp/issues/2410. Now, we have reached this limit five days ago, too, so no more Travis builds can start at the rl-uc3m org.

I'm afraid it's time to say goodbye to Travis CI.

[PeterBowman]

@jgvictores has found https://github.com/cyberbotics/webots-animation-action. GH Actions seems such a powerful tool!

[PeterBowman]

Useful links:

• https://docs.github.com/en/actions
• https://docs.github.com/en/actions/reference
• https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
• https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions
• https://docs.github.com/en/actions/reference/environment-variables#default-environment-variables
• https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#supported-software
• https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits
• https://docs.github.com/en/actions/learn-github-actions
• https://docs.github.com/en/actions/learn-github-actions/migrating-from-travis-ci-to-github-actions

[PeterBowman]

Enabled in color-debug: https://github.com/roboticslab-uc3m/color-debug/commit/76a82da9b5f900b1cd9e146fc863e96d5ce4230d.

For further reference:

• Define env variables in conditional steps per OS instead of using a strategy matrix and custom job names: https://github.com/roboticslab-uc3m/color-debug/commit/ad177c4733a943d0594fffb9223fe0052a005060.
• Split main job into many simple jobs for improved concurrency: https://github.com/roboticslab-uc3m/color-debug/commit/24c64826c8fcb20cd2943bc8195bae9d506ac73f. I am not merging this since YAML file size and complexity increases a lot as well as total workflow duration. See GHA run and the resulting artifact storage.

[PeterBowman]

Default CMake version in Ubuntu bionic and focal images is 3.19. YARP/YCM require at least 3.12. To stay on the safe side, I'm going to force builds against CMake 3.12 with https://github.com/jwlawson/actions-setup-cmake (we usually work with newer versions nowadays, which is prone to introduce non-back-compat behavior). Note conventions have changed and modern CMake projects can be configured, built and installed (docs) with:

cmake -S path_to_source -B path_to_build # creates directory structure if necessary
cmake --build path_to_build
cmake --install path_to_build

Pre-CMake 3.13 projects need to change the working directory and have the build dir created before first config (docs):

cmake -E make_directory path_to_build
# you can use `cd` here or prepend `cmake -E chdir` to each command as shown below:
cmake -E chdir path_to_build cmake ..
cmake -E chdir path_to_build cmake --build .
cmake -E chdir path_to_build cmake --build . --target install

See https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/780d25a92db5a1dc58b541bc3d470142f7874a8b.

[PeterBowman]

Step commands (jobs.<job_id>.steps[*].run) are executed in the default OS-shell (Windows: PowerShell, non-Windows: Bash) unless specified otherwise via jobs.<job_id>.steps[*].shell. The bash option is also available on Windows systems using the bash shell included with Git for Windows. If all steps in one or all jobs are meant to be executed on both Windows and non-Windows systems, adding these lines instructs the runner to always prefer bash so that there is no need to add a shell property per step:

defaults:
  run:
    shell: bash

See example at color-debug. This default value may target all steps in one job (docs) or all steps in all jobs (docs).

I'm not sure how portable Bash really is, but it seems to work for our simple use-case in color-debug (Linux+Windows). An alternative, multi-platform strategy involves using CMake as a shell command launcher via cmake -P {0} (ref). This solution has been adopted in YARP (ref).

[PeterBowman]

Commit https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/4cf5926b4a5172946cf9ff945779f63efba93cf5 introduces caching of main dependencies (YARP and KDL) with ccache (check also this article). Comparing build times:

• Not cached (run): ~6 minutes for YARP, ~1 minute for KDL.
• Cached (run): 29 seconds for YARP, 2 seconds for KDL.

I am using https://github.com/hendrikmuhs/ccache-action, which internally implements this methodology using the official cache action (guide).

The following CMake options enable ccache for selected builds:

cmake .. -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache

Note: this is a CMake variable, CMAKE_<LANG>_COMPILER_LAUNCHER. In CMake 3.17, an environment variable of the same name was introduced (ref).

Speaking of kinematics-dynamics, I am enabling ccache only for YARP and KDL. Also, these builds are benefiting from parallelization via cmake --build <dir> --parallel (uses default parallel level if no value is provided). Intentionally avoiding parallel builds of YCM due to numerous failures experienced in past Travis jobs. I am not boosting the main build (i.e. kinematics-dynamics) on purpose, might change my mind later on.

PS: added -DYARP_DISABLE_VERSION_SOURCE=ON to YARP build config in order to "avoid rebuilding everything every commit" (ref).

[PeterBowman]

It turns out scheduling jobs the way we did with Travis is a but more clumsy on GHA. It is not possible to evaluate conditions when defining an inclusion in the job matrix (if event_type is schedule then add job). All combinations for scheduled and non-scheduled event types must be defined at once, then we'd just exclude YARP master branch combinations for non-scheduled jobs. However, jobs.<job_id>.if cannot be used here since the matrix is defined after evaluating the condition (ref, https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/ebddfcc131056568a7ae7b8656f92564b2c85ee7).

A dummy matrix key that depends on the event type can be used to determine exclusions, though, as explained here: https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/038a706bd4a03895da68cd9691850dca8b88afed. Note it is not possible to apply a similar trick for conditional job inclusions: https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/fa3d1f579824472f328e3f6bf62746f4bc0f6143.

Alternatives involve using a dynamic matrix which is conditionally defined in one job and actually consumed in another, dependent job: ref. IMO even more clumsy. Of course I could just avoid all this mess and always run jobs on YARP's master branch, i.e. on every push, and possibly set jobs.<job_id>.continue-on-error to true so that the workflow run passes regardless of errors (which I hope would be notified somehow).

PS: according to docs, "scheduled workflows run on the latest commit on the default or base branch", so I can't fully test this unless my feature branch is merged.

[PeterBowman]

The amor-api private repo now participates in the CI workflow at https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/201fd98fd4a1132633aecbea93596cd504e73f38 (WIP). In order to grant access, I am providing my personal access token (PAT) via the org-wide secret AMOR_API, currently enabled only on repos that need to fetch amor-api sources (kin-dyn, yarp-devices and amor-main). More info:

• https://docs.github.com/en/actions/reference/encrypted-secrets
• https://docs.github.com/en/actions/reference/authentication-in-a-workflow
• https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token
• https://github.community/t/best-way-to-clone-a-private-repo-during-script-run-of-private-github-action/16116/7
• https://docs.github.com/en/actions/learn-github-actions/sharing-workflows-with-your-organization#sharing-secrets-within-an-organization

[PeterBowman]

Similarly to Travis CI, GitHub Actions workflow runs can be skipped if the commit description includes a [skip ci] string.

Quoting https://github.blog/changelog/2021-02-08-github-actions-skip-pull-request-and-push-workflows-with-skip-ci/:

GitHub Actions now supports skipping push and pull_request workflows by looking for some common keywords in your commit message.

If any commit message in your push or the HEAD commit of your PR contains the strings [skip ci], [ci skip], [no ci], [skip actions], or [actions skip] workflows triggered on the push or pull_request events will be skipped.

[PeterBowman]

Speaking of kinematics-dynamics, I am enabling ccache only for YARP and KDL. Also, these builds are benefiting from parallelization via cmake --build <dir> --parallel (uses default parallel level if no value is provided). (...) I am not boosting the main build (i.e. kinematics-dynamics) on purpose, might change my mind later on.

Since ccache seems reliable enough, I have enabled caching for the main build, too. Not using --parallel for better browsing of the compile log.

[PeterBowman]

Alternatives involve using a dynamic matrix which is conditionally defined in one job and actually consumed in another, dependent job: ref. IMO even more clumsy. Of course I could just avoid all this mess and always run jobs on YARP's master branch, i.e. on every push, and possibly set jobs.<job_id>.continue-on-error to true so that the workflow run passes regardless of errors (which I hope would be notified somehow).

I am not the only one who seeks this feature: community post, https://github.com/actions/runner/issues/2347. The continue-on-error seems to prevent a specific job from canceling other jobs, but still marks the whole run as failed. I was expecting some sort of notification about allowed failures as other users report in the linked issue (this was a Travis feature). In contrast, fail-fast defines the default behavior for all jobs, which is basically the same thing as continue-on-error with a different scope. See also https://github.com/actions/runner/issues/2347.

Why would we want this? For early testing of experimental upstream dependencies (i.e. YARP/YCM). Options:

• (current status quo) Build pushes and pull requests against stable robotology branches (8 jobs AToW). On schedule (once per week), trigger an automated build and add more jobs targeting unstable (master) YARP/YCM branches (totalling 12 jobs). Who is notified when a scheduled run fails? According to docs:

  Notifications for scheduled workflows are sent to the user who initially created the workflow. If a different user updates the cron syntax in the workflow file, subsequent notifications will be sent to that user instead. If a scheduled workflow is disabled and then re-enabled, notifications will be sent to the user who re-enabled the workflow rather than the user who last modified the cron syntax.

• Always build against stable and unstable robotology branches. In the current matrix configuration, this would result in 12 jobs per workflow run. Note GH Actions for free plans is limited to 20 concurrent jobs in the personal/organization account (ref). I understand that if two runs are started on two repositories in our org, there will be 20-12+12=4 jobs awaiting the completion of a different run. In this case, failure notifications are sent to the committing user.

A dummy matrix key that depends on the event type can be used to determine exclusions, though, as explained here: roboticslab-uc3m/kinematics-dynamics@038a706.

This workaround doesn't cope well with JSON-based matrix entries: https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/5b2e3d8c9daa3b436aebf9c41de90a9f27b33ffe. The run got stuck and I had to cancel it. This is a rather lame reason in favor of switching to the second option (always 12 jobs), but still...

As spoken with @jgvictores, we'll stick to old behavior, i.e. option 1, i.e. only build against unstable branches on schedule. The issue was fixed at https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/98179d6a7ca03c242e4f95b02368d5e88fdba7e5, turned out to be valid YAML syntax, yet contrary to the actual intent.

[PeterBowman]

Commit https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/59c6b9a5c2d9e5cbfdb9d34c147f98de8fe0cd5c enables path exclusion on push events in such a way that a new run is not triggered whenever the algorithm detects that no file outside the doc/ directory has been changed, also ignoring all .md files.

Docs:

• https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths
• https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet

[PeterBowman]

Mostly ready at kinematics-dynamics: ci.yml. Cron jobs have been configured in a weekly fashion, on Sunday, 02:00 UTC. Also, I have prepared a template for quick reuse in other repos: roboticslab-uc3m/.github, docs.

[PeterBowman]

TO-DO list (from https://travis-ci.com/github/roboticslab-uc3m):

• [x] yarp-devices (weekly cron: Monday)
• [x] kinematics-dynamics (weekly cron: Tuesday)
• [x] tools (weekly cron: Wednesday)
• [x] speech (weekly cron: Thursday)
• [x] openrave-yarp-plugins (weekly cron: Friday)
• [x] vision (weekly cron: Saturday)
• [x] teo-bimanipulation (monthly cron)
• [x] teo-self-presentation (monthly cron)
• [x] follow-me (monthly cron)
• [x] asibot-hmi (monthly cron)
• [x] teo-main (no cron)
• [x] amor-main (no cron)
• [x] asibot-main (no cron)
• [x] color-debug (no cron)
• project-generator (CI disabled)
• xgnitive (CI disabled)
• Dextra (CI disabled)
• gait (CI disabled)

Also:

• [x] Disable Travis CI (unlink from GH org account?) -> suspended
• [x] Delete amor-api deploy-key (the setup was explained at https://github.com/roboticslab-uc3m/amor-main/issues/3#issuecomment-373910089)

[PeterBowman]

We could also migrate the generation of Doxygen documentation and GitBook manuals (currently at robots.uc3m.es) to GH Actions. For instance, this script was used in robotology/gazebo-yarp-plugins to generate http://robotology.github.io/gazebo-yarp-plugins/master/.

Links:

• https://docs.github.com/en/github/working-with-github-pages
• https://github.com/marketplace?query=github+pages
  • https://github.com/JamesIves/github-pages-deploy-action
  • https://github.com/crazy-max/ghaction-github-pages
• https://github.com/jgvictores/wikirobots/blob/master/scripts/update-daily.sh (permalink)

Additional tools that might come in handy:

• https://github.com/fkirc/skip-duplicate-actions
• https://gist.github.com/jasonrudolph/1810768 + https://github.community/t/17887/4 (throttle jobs on a per-day basis)

[PeterBowman]

Three repositories are currently configured to run weekly CI builds: tools, speech, kin-dyn.

Remarks:

• Scheduled runs are not timely. Individual jobs may start even half an hour after the appointed time, also they don't seem to be started all at once.
• GitHub removes cached files if not used after 7 days (ref). I wonder how precise/strict is this. I have found that kin-dyn cache was restored after one week plus 3 minutes: ref (enable "Show timestamps"). Sadly, there is no way to inspect nor delete cache files, yet (https://github.com/actions/cache/issues/2).

[PeterBowman]

This is nice: the .md filter prevented https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/a9acac8b8246c77be885f83648b12eddcac43d3c from triggering a CI run, but the weekly cron workflow started nonetheless (run).

I see that cache eviction is not happening on weekly crons. I think it's safe to keep on migrating to GHA on the remaining repos. However, I'm probably dilating the no-build period on crons for selected, low-traffic repositories listed at https://github.com/roboticslab-uc3m/questions-and-answers/issues/91#issuecomment-784199905.

[PeterBowman]

I have integrated https://github.com/fkirc/skip-duplicate-actions in our CI workflow: https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/55cfdcc78f53b71da69f746cfa84b15e6483a2ea. It covers the following use cases:

• Skip duplicate workflow-runs: don't trigger a run twice if it had been already completed on a feature branch before merging. For instance, let's assume we pushed and run CI of commit X on branch Y, then Y is merged into master to produce commit Z. CI jobs are skipped if Z was either a fast-forward commit, or a merge commit on top of master. If branch Y has outdated commits (i.e. some changes were pushed to master before the final merge of Y), then the run for Z is triggered as expected.
• Cancel outdated workflow-runs: cancel an ongoing run if more changes have been pushed to the same branch.

I believe these features will be eventually integrated into vanilla GHA. On the other hand, I don't quite understand "Skip ignored paths". I think GHA already provides backtracking, or at least the following scenario has been successfully tested:

• Start branch my_feature, commit a change in C++ code, push and await CI completion.
• Commit more changes to C++ code, but don't push yet.
• Commit a change to any .md file, commit and push. Note **.md was configured in paths-ignore.

I thought GHA would skip the workflow run for the last action because of "stupidly looking at the current commit", as stated in the README for skip-duplicate-actions. It didn't do so, though. I think it's performing a two-way comparison on each push: ref.

[PeterBowman]

We could also migrate the generation of Doxygen documentation (...)

Done in kin-dyn at https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/aed50de49991aceae784fb09f9ef3bd043623742 using https://github.com/crazy-max/ghaction-github-pages (doxygen.yml, https://roboticslab-uc3m.github.io/kinematics-dynamics). I was considering two ways to do this:

• Add a new job to the CI workflow, conditionally enabled in master branch. In short: triggered on every push to master.
• Add a new workflow that would trigger either on every push to master, or periodically via cron.

I've chosen the latter and triggering on every push as long as GH detects changes on selected directories. I am also enabling manual dispatch, i.e. new runs can be manually triggered via GUI (see "Run workflow" button here).

Notes:

• Nuked main.dox in favor of README.md as the main page for doxy.
• I resorted to using absolute paths pointing at GH CDN for images to work both in GH markdown and in Doxygen markdown: https://github.com/roboticslab-uc3m/kinematics-dynamics/commit/63871757390c48ab6f33b77a902e09a0d914e25c.
• We started generating our Doxyfiles via CMake thanks to YCM at https://github.com/roboticslab-uc3m/questions-and-answers/issues/79. This decision was tailored to our robots.uc3m.es server because of working with obsolete packages (Ubuntu 15.10), therefore YCM allowed us to build and use CMake 3.5, yet take advantage of modern FindDoxygen functions introduced in CMake 3.9. Note recent releases of Doxygen actually require a fix introduced in CMake 3.13 (also handled by YCM: https://github.com/robotology/ycm/pull/249). Since we are definitelly dropping robots.uc3m.es for documentation generation, I'm raising the requirements on doc/CMakeLists.txt to CMake 3.9 for stand-alone doxy builds (ref).
• Had to tweak a few Doxygen options, see https://www.doxygen.nl/manual/config.html.
• Doxygen files are deployed on the "gh-pages" history-less branch, force-pushing every new commit.

Occurrences of update-dox.sh across our org, currently active per update-daily.sh:

• [x] kinematics-dynamics
• [x] speech
• [x] tools
• [x] vision
• [x] yarp-devices
• [x] openrave-yarp-plugins

Ignoring these per lack of "doxygen" link in repository-index.md: follow-me, gaitcontrol (update script outside project), force-torque-balance, xgnitive.

See search results for remaining occurrences.

[PeterBowman]

We could also migrate the generation of (...) GitBook manuals (...)

Done at https://github.com/roboticslab-uc3m/teo-developer-manual/commit/c9a99f03dcace6681d8c0d17fff6b33ab54b2dfc (gitbook.yml). Initially, I resorted to installing Calibre and GitBook from scratch (node modules + plugins), which took +3 minutes to complete: gitbook.yml. To avoid this, I tried to take advantage of custom Docker containers, thus cutting run times down to one minute (but no PDF). Links:

• https://github.com/billryan/docker-gitbook (DockerHub, MicroBadger)
• https://hub.docker.com/_/node (using node:10.24-alpine)
• https://calibre-ebook.com/download_linux
• https://docs.docker.com/develop/develop-images/multistage-build/
• https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
• https://docs.github.com/en/packages/guides/container-guides-for-github-packages
• https://github.com/roboticslab-uc3m/docker-gitbook
• https://github.com/orgs/roboticslab-uc3m/packages/container/package/gitbook
• https://github.com/roboticslab-uc3m/teo-developer-manual/deployments

Sadly, PDF generation is broken, so we've decided to take it out (along with MOBI and EPUB files). After installing hundreds of MBs of dependencies (Calibre/ebook-convert + svgexport/puppeteer/Chromium) and solving a known --no-sandbox issue (see intructions, permalink), it gets stuck in the gitbook pdf step (but no errors are thrown). This is the closest I have gotten, aditionally it is necessary to create a new user and assign permissions (see linked puppeteer's troubleshooting guide):

FROM alpine:latest AS download_calibre
ENV CALIBRE_DOWNLOAD_URL https://download.calibre-ebook.com/5.13.0/calibre-5.13.0-x86_64.txz
RUN mkdir /tmp/calibre && \
    wget --no-check-certificate -qO- $CALIBRE_DOWNLOAD_URL | tar xvJ -C /tmp/calibre

FROM node:10.24-buster-slim
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/opt/calibre/lib
ENV PATH $PATH:/opt/calibre/bin
COPY --from=download_calibre /tmp/calibre /opt/calibre
RUN npm install -g --unsafe-perm \
        gitbook-cli \
        gitbook-summary \
        svgexport && \
    gitbook install && \
    rm -rf /tmp/* && \
    npm cache clean --force
EXPOSE 4000
VOLUME /gitbook
WORKDIR /gitbook
CMD ["gitbook", "--help"]

TODO per update-daily.sh:

• [x] teo-developer-manual
• [x] developer-manual
• [x] installation-guides

PS the old robots.uc3m.es/gitbook-xxx/ addresses can be easily redirected (HTTP 301) with something like this in /var/robots/htdocs/.htaccess (tutorial):

RedirectMatch 301 "^/gitbook-teo-developer-manual/(.*)" "https://roboticslab-uc3m.github.io/teo-developer-manual/$1"

[jgvictores]

Note: https://github.com/asrob-uc3m/repostatistics

• http://robots.uc3m.es/contributors
• http://wiki.asrob.uc3m.es/contributors

[jgvictores]

@PeterBowman THANK YOU SO MUCH!!!! 🎉🎉🎉🎉🎉

[PeterBowman]

I have just learned that "scheduled workflows are automatically disabled when no repository activity has occurred in 60 days" (ref). The following notice is displayed right now in the Action's tab of the speech repo:

If we didn't care about the future of the planet, then we could schedule a cron on our robots.uc3m.es server to run gh workflow enable ci every now and then, see docs.

[PeterBowman]

Note sometimes builds may fail for no apparent reason. We have recently seen pretty weird segfaults on unit tests affecting just Clang builds on YARP master and Ubuntu bionic, among other issues. It is fine, probably ccache artifacts are conflicting on linkage or something like that. Don't panic and invalidate the cache like this: https://github.com/roboticslab-uc3m/tools/commit/38b02d2b807e91a4009b460ac6973289da6cc004. Make sure to also target master builds, then manually trigger the workflow again to make sure it's fixed and rebuild the cache. Currently there is no other way to clear the cache (GH Support).