How to use this for external people #3

Open ranocha opened 1 year ago

This is a great project! I'm really excited about it - and would like to use this kind of infrastructure since I'm one of the principal developers of Trixi.jl.

I just made a first PR to see how this works in practice: #2. When I click on "Details" next to the CI job "ci/gitlab/git.rwth-aachen.de Pending — Pipeline running on GitLab", I am redirected to the website https://git.rwth-aachen.de/users/sign_in telling me to sign in to the RWTH Aachen Gitlab. How shall I proceed as external developer?

CC @Atrisan @felilxtomski @sloede

If I sign in with GitHub, I get a 404 error at https://git.rwth-aachen.de/trixi-framework/TrixiTestCI.jl/-/pipelines/1105278 (linked from the last successful run on main).

The project at https://git.rwth-aachen.de/trixi-framework/TrixiTestCI.jl is still private, so externals cannot view the results. I am currently looking into a proper configuration.

When the project is visible you should be able to see the results without any login.

The PR pipeline was triggered properly in our Gitlab instance, I am not quite sure why it is not shown properly, but this might only be a matter of time until it is available.

Hm. The main reason for us making the project private was that the RWTH GitLab seems to have a crazy high SEO ranking for Google. That is, when we had the Trixi.jl mirror public, searching for Trixi.jl led users to the mirror website and not to the original repo anymore. Since, by definition, a mirror has exactly the same content as the main repo, this was an issue, since it led many people to the wrong place.

Is there a way to get around this?

So, after some testing there are new insides:

It is possible to have multiple pull mirrors in GitLab, but the latest pipeline will overwrite the visible results in Github, i.e., the pipeline in my mirror is visible in GitLab, but @sloede your pipeline is invisible from Github. With different configs that might be possible, but I am still testing. The internal visibility does also not help, since people would at least require guest access to the repo to view the CI/CD.
My mirror should be visible without a login and almost all features are deactivated, but the repository and code is coupled to the CI/CD, You can however rename the repo into something cryptic. I cannot estimate the effects on the SEO rankings, because projects were unlisted for the longest time.

If the visibility is required by a limited set of core developers, it might be feasible to use the GitHub login and add the People required to see the results to the GitLab repo by hand.

Interesting, thanks for the explanation

Update:

Trixi is working on our cluster and we now provide modules for Julia (Version 1.8 and 1.9) so you won't need to build it beforehand. (Just not the Julia-executions I was trying to write)
To simplify the scoping and limit the access from externals, only protected branches will be allowed to run the mirrored CI.
- Anyone with write-access to the protected branches will be able to trigger the CI
- This would also include merge requests, from those protected branches
To properly view the results, the Gitlab repo needs to be public
The corresponding service account only provides 200 core hours per month (which is not much) for more a compute project would be necessary: https://help.itc.rwth-aachen.de/service/rhr4fjjutttf/article/45825b06afb647e194be4a5b9f5b8768/
Everything in the Gitlab CI would run on the cluster, the resources/jobs can be configured via variables in the YML file adapting the internal Slurm parameters
- A basic documentation for the Variables can be found here: https://git-ce.rwth-aachen.de/adrianschmitz2/runner-mirror/-/wikis/Modes-and-Variables
- The Slurm parameters are available in their documentation: https://slurm.schedmd.com/sbatch.html

Would that work for Trixi and its development? @sloede @ranocha

So this means we cannot run any CI in standard PRs on GitHub? That's a severe restriction. Would it be possible to have something like the option to run it from a PR if a maintainer triggers it explicitly?

Is it public already? We had some troubles with a public mirror of Trixi.jl earlier, since all Google searches made it the first entry so that people couldn't find our real repo anymore

Currently it is not public. Depending on whether only devs need to be able to see the results, it would be possible to add them to the repo. Alternatively, the visibility of the mirror could be set to Internal then the mirror should not be listed in search engines, but the GitHub Login at our GitLab instance could be used to login as a guest and view the results. Currently Michaels mirror is within a group, which overwrites the settings of the internal repository.

For the PRs it depends on where they come from. In the non-restricted case all branches of the project can trigger a pipeline (forks cannot though). With regards to security only "approved" people should be allowed to trigger the pipeline and run code on the cluster. Since I am not to sure about the access rights for branch creation in GitHub I am starting from there. If it is not possible for a random person from the internet to create a branch, which is very likely to be the case, it will be possible to start pipelines from any branch, including pull requests. I still need to do some basic pen-testing beforehand.

Internal visibility sounds great!

We have restricted access for people to create branches in the repo itself. Whoever can do that should be trusted. Are there any ways to run tests for PRs from forks (the majority of contributions)? Or do we need to believe the tests on GitHub runners first, merge the PRs, and trigger new tests afterwards?

Branch creation is limited to people with direct access, so making the CI available for all branches should not be a problem. PRs from forks will not automatically trigger the GitLab CI, which from your perspective might be non-optimal, but would have been a reason for shutting GitHub support down for the RWTH. You can however first merge into a local testing branch, to trigger the CI on the HPC cluster and then from the testing branch into main.

Internal visibility sounded to good to be true, apparently users with a GitHub login do not technically count as logged in, so they won't be able to see the pipeline unless it is public (contrary to my expectation) or they were granted access to the mirror, which would be the exact same for a private mirror.

So we're basically limited to access only for Trixi core devs, correct?

Yes, for triggering the pipeline that is both a technical and a legal problem. For the visibility of the pipeline I will have to contact the people responsible for the GitLab instance, they might be able to tweak some things, so that the results can be publicly visible without listing the repository on any search engine.

Thanks a lot!

trixi-framework / TrixiTestCI.jl

How to use this for external people #3