[Infra] Wishlist for GitHub actions

eseiler commented 4 years ago

Description

This is a list of all the things we want to add to the GitHub actions.

Wishlist

[ ] https://scan.coverity.com/faq#what-is-coverity-scan
[x] Add macOS to SeqAn3. PR
[x] Add jobs that build with SeqAn2 ~but do not fail the CI if the job fails~. PR
[x] Cancel old workflows. PR
[x] Cache external dependencies. PR
[x] Limit execution to 2h. PR
[x] Add hierarchical caching. PR
[x] Reduce log verbosity. PR
[x] ci skip. natively
[ ] Only run macOS tests after Linux tests succeeded.
[ ] Do not fail if the caching fails (See example)
[x] Put docs on a dedicated website instead of just offering it as artefact (and delete from website if PR is closed/merged). See https://vercel.com/ PR
[ ] Report to CDash.
[ ] Set RAM Limit for each compiler: A test should not need more than 2Gb of RAM (ideas: https://unix.stackexchange.com/a/44987)
[ ] Add a mechanism to build the complete test suite before a merge and do only a minimal subset during the review process (like gcc 7-10 linux unit tests). Since benchmarks, coverage, documentation, snippet, acyclic tests are not so important during the review process.
- The workflow would change to:
- create PR
- get basic feedback that basic compiler work
- 1st review until approval
- 1st review approves
- 2nd review until approval
- 2nd review marks PR as ready-to-merge
- full test suite
- while full test suite fails: fix
- full test suite succeeds
- 2nd review approves
- merge

Tasks

[ ] Pick a feature.
[ ] Fulfil DoD for feature.

Definition of Done

[ ] Implementation and design approved.
[ ] CI passes and behaves as expected.

marehr commented 4 years ago

Can you link the PRs that already completed some of these?

eseiler commented 4 years ago

I added related PRs

eseiler commented 4 years ago

Regarding the RAM limits: We currently have some tests that need around 3 GiB for compilation. (How to measure) If we don't want to / cannot do it via the OS, we can just measure RAM within the coverage build (Debug should use the most RAM), and fail if it's more than 2048 MiB.

Anyway, for a 2 GiB limit, we would need to reduce the RAM usage of the tests. The big advantage would be that the travis nightlies for the different architectures would then work properly. If the architecture is not amd64, the runner only have around 4 GiB of RAM, so not quite enough to run make -j2, but make -j1 would result in a timeout....

smehringer commented 4 years ago

@eseiler Does Github actions feature something like the [no ci] tag we had for travis? it would be great when applying suggestions that we can skip the CI for intermediate commits.

eseiler commented 4 years ago

I already had a look at it once, we can try different stuff: https://github.community/t/github-actions-does-not-respect-skip-ci/17325 Main problem is that push and pull_request may have different commit messages; but I can try it "locally"

But for simplicity we would need to decide for one tag ([ci skip], [skip ci], ???)

smehringer commented 4 years ago

I already had a look at it once, we can try different stuff: https://github.community/t/github-actions-does-not-respect-skip-ci/17325 Main problem is that push and pull_request may have different commit messages; but I can try it "locally"

But for simplicity we would need to decide for one tag ([ci skip], [skip ci], ???)

this is not urgent but would be nice :) You can decide for the tag.

marehr commented 4 years ago

@eseiler Does Github actions feature something like the [no ci] tag we had for travis? it would be great when applying suggestions that we can skip the CI for intermediate commits.

Is that really a problem though? A new commit / push will abort an existing build, right? If someone adds [no ci], but when we do a squash merge, we might accidentally merge something that we don't want to merge.

eseiler commented 4 years ago

Is that really a problem though? A new commit / push will abort an existing build, right? If someone adds [no ci], but when we do a squash merge, we might accidentally merge something that we don't want to merge.

Yeah, but then you need to immediately push a new commit and when you apply suggestions, you might not want to do this right at this moment.

ci skip is one of the features that all big CI providers except github provide. It's not necessary, but it would be nice...

Also, we do not do rebase merges, so we should always have a merge commit ("Merge into master" or something). Anyway, we can just exclude push events from the ci skip rule

I have some implementation: https://github.com/eseiler/seqan3/pull/4/files We can decide if we want this or not, but at least we know how it would look like. I have to admit that this introduces complexity (the one job that needs to be run to get the commit message) ... so there is more stuff that might break.

eseiler commented 4 years ago

There is a fork/PR for the cache action which allows to update caches: https://github.com/actions/cache/pull/353 We could use this patch to update the caches instead of always uploading a new one. If we have 10 jobs per run with each around 50MiB of cache, we can store at most 10 runs (10*500 = 5GiB which is the cache limit). If we instead updated the caches, we could store 10 branches instead of 10 runs ...

marehr commented 3 years ago

I looked a bit into automatic docs preview

Some general things:

We want to deploy user doc (7,805 items, totalling 171.9 MB) and dev doc (10,423 items, totalling 207.3 MB)

TODO

surge.sh:
- it rejected my dev doc push
vercel.com:
- Looking at https://vercel.com/pricing it seems that we need to deploy it via a personal account.
- It seems like you can apply as a OpenSource project:
- For Open Source Projects: Submit an open source project for free or discounted resources from Vercel.
- See their limits https://vercel.com/docs/platform/limits
- It has a preview option: see https://github.com/marehr/docs.seqan.de/pull/1
I tried to host our documentation ourselves:
- see https://docs.seqan.de/seqan/preview/seqan3-pr1-dev/index.html
- I could get the filesize down to
- 38M seqan3-pr1-usr (~73% smaller)
- 47M seqan3-pr1-dev (~77% smaller)
- ~100Mb per PR preview
- This uses a trick of pre-compressing all files with gzip / brotlin and serving those files directly.
- That means my_graphic.svg will be stored as my_graphic.svg.gz
- If the browser requests the resource my_graphic.svg it will look if my_graphic.svg.gz exists and if it does, the web server serves it instead with a Content-Encoding of gzip / br.
- This needs a real web-server like apache/nginx.
- See https://blog.desgrange.net/post/2017/04/10/pre-compression-with-gzip-and-brotli-in-apache.html for an apache
- Brotlin can compress our complete documentation to about ~16MB, but takes 328.63s. (In comparison gzip takes 6.08s compress to ~33MB)
- so mounting that on the fly would also be possible, but that would need some custom-webserver infrastructure
- I also thought about using a git repository where we dump every snapshot into it's own branch.
- git has the property that it will compress data in the background and that it stores the same file (deduplication) only once
- The problem here was again, that we would need a custom-webserver that "checks out" the requested file on demand and serve it.

eseiler commented 3 years ago

Do you know why surge.sh rejects the dev doc push? Didn't immediately find limits on the website? Was there an error? It's probably the file size?

marehr commented 3 years ago

Do you know why surge.sh rejects the dev doc push? Didn't immediately find limits on the website? Was there an error? It's probably the file size?

I'm not completely sure why, but it refused it. I tried it a second time and searched for the error, but couldn't find anything. So I didn't look into it any more.

In the meantime I got vercel to build our documentation with a preview https://docs-seqan-de-kvk88zba5.vercel.app/

eseiler commented 3 years ago

For Open Source Projects: Submit an open source project for free or discounted resources from Vercel.

https://vercel.com/knowledge/can-vercel-sponsor-my-open-source-project

eseiler commented 3 years ago

ci skip is now natively supported: https://github.blog/changelog/2021-02-08-github-actions-skip-pull-request-and-push-workflows-with-skip-ci/

eseiler commented 3 years ago

Cron jobs create quite a lot of issues:

Create issue in product_backlog
one issue and do comments + (close/re-open)

seqan / product_backlog