Move the Perlang project hosting to self-hosted GitLab instance

[perlun]

Moved to GitLab

Please continue to the new version of this issue here: https://gitlab.perlang.org/perlang/perlang/-/issues/423. The GitHub page you are currently reading will not contain the latest information on this issue.

---

At some point, I believe we should move the Perlang hosting to a self-hosted GitLab instance, for a few different reasons:

• Better autonomy. As the Debian project describes it:

  This account is not intended to serve as the canonical (specified with Vcs-* fields of debian/control) location for corresponding Debian source packages. Most often such repositories should be made available on the Debian project's public forge Salsa to guarantee autonomy.

  "Autonomy" in the previous sentence links to this page: https://web.archive.org/web/20081002034726/http://autonomo.us/2008/07/franklin-street-statement/

  Using GitLab means we, as a project, will have full control over the repository. GitHub outage, because of problems with their hosting? Forget it. Of course, we can instead have "our own" outages, but still, this give us more control over a critical part of our own infrastructure.

• More freedom. We are currently relying upon GitHub (which is a proprietary, SaaS-hosted software). The free (as in freedom) version of GitLab, namely GitLab Community Edition is licensed under the MIT Expat license, a license which qualifies as Free Software by the FSF and Open Source by the OSI.

• Be able to use GitLab CI. GitHub Actions work quite well, but it does have certain disadvantages. One of the things I particularly dislike is the "Status checks that are required" which is configured via "branch protection rules" under https://github.com/perlang-org/perlang/settings/branches. Here, the required status checks are configured. There's just one problem with this: it makes the CI settings configured in the web interface be tightly coupled to the action names defined under .github/workflows. You rename one of these, and the CI settings break. I find this rather fragile and I think the GitLab CI approach - where much more of this is defined in .gitlab-ci.yml is much preferable.

  • Using GitLab CI with GitHub can be done though, but:
    1. It requires GitLab Premium or Ultimate (i.e. the paid-for tiers)
    2. It still doesn't free you from the obnoxious "Status checks that are required" problem described above.

  In general, I think GitLab CI has more features (caveat: I've only used the commercial subscription options for GitLab thus far; the Community Edition is more limited by nature, even though GitLab (the company) does their best by e.g. stating that "[the] majority of new features made by GitLab Inc. will be open source". We should probably set up a "public pilot" for this an try it out (on our own/self-controlled hardware), to ensure that the feature set matches our requirements.

• Have control over our own CI log retention times. GitHub only lets us retain logs for 90 days, which is fine for many cases but what about the ones where it isn't? What if we want to view the logs for something that happened 6, 12 or 18 months ago? Relating back to the "autonomy" point above, hosting our own GitLab instance gives us much better control over settings like this.

Why not just host it on gitlab.com?

This would clearly be an option, but:

• This won't give us full autonomy.
• If we still want to host it ourselves at some point, we will be updating URLs multiple times which will be inconvenient.
• Public projects on gitlab.com has a limit of 400 compute minutes per month (https://about.gitlab.com/pricing/#compute-minutes). This can easily become a problem. We can request to be part of the GitLab Open Source program, which increases this limit greatly; it has to be renewed and approved by GitLab yearly.
• GitLab.com has CI runners only for Linux at the moment, with Windows and macOS support in Beta (2023-11-27). If we host our own (on e.g. Azure), we can run Linux and Windows with GitLab Runner. macOS will be more challenging (we need to run it on our own hardware or find a cloud-provider which will sponsor us with macOS cloud VMs). Maybe the latter will be doable when we are ready for this.

Having that said, moving to gitlab.com as a first step would mean that we are better prepared for one day hosting it ourselves. We will have a CI configuration which already works out-of-the-box on GitLab CE (unless we use any EE-only features), so in this sense moving to gitlab.com would be a step in the right direction Also, it would be a smaller step than doing a "full" migration to a self-hosted instance.

Other options

• Switch to Gitea. This is what Jeff Geerling has done.
  • Freely available under the MIT (Expat) license, similar to GitLab CE.
  • Seems to use a CI system similar to GitHub Actions though, so we can't really take advantage of the .gitlab-ci.yml approach if we go this route.
  • Importing from GitHub doesn't seem to be supported; importing the Git history is obviously easy, but doing a full import including issue and pull request history doesn't seem to be supported out of the box. Compare this to GitLab, where the import seems to cover much more: https://docs.gitlab.com/ee/user/project/import/github.html
• Switch to Codeberg
  • More limited CI, or rather, CI with a manual on-boarding process. Not yet investigated whether the capabilities provided by Woodpecker CI is on par with GitLab CI and/or GitHub Actions.

Hosting options

• Host it in a public cloud; pay for it or try to get sponsored hosting somewhere.
• Host it ourselves, and pay for colocation or a business Internet connection + hardware.

Both of these options means that we will need to:

1. Take care of maintaining the OS and perform OS upgrades as necessary.
2. Upgrade GitLab regularly, to get new features and security updates in place. This is not a very complex process, but it still means a little bit of work every month when a new version is available.
3. Take care of operations ourselves. For example, the ISC GitLab has had to put measures in place because of a high volume of spam accounts being created. We can of course mitigate this to an extent by limiting/preventing new user accounts from being created, but that risks to unnecessarily raise the bar for contributors.
   • This an argument in general against migrating away from GitHub, for that record. Many people have a GitHub account and can easily jump in to a discussion. Not many people will have a Perlang GitLab account, from day one. But could this also possibly be a positive thing sometimes? It it always just bad that people have to put in a little bit of effort before jumping in to a random discussion...?

[perlun]

If/when we do this:

• We should close all issues here, to make things less confusing. Perhaps even delete them altogether? (but that breaks web history a bit) Or at least lock them from further comments.
• We keep the repo here as a push-mirror (or even two-way mirror if we are brave and GitLab supports it?) of the self-hosted GitLab instance.
• We should add a new, pinned issue with a link to the new project URL, to make it easy for people to find the new home for the project.

[perlun]

Deciding on the host name

One of these would make most sense:

• git.perlang.org
• gitlab.perlang.org

Real-world examples

(inspired by https://wiki.p2pfoundation.net/List_of_Community-Hosted_GitLab_Instances)

• https://salsa.debian.org
• https://gitlab.gnome.org/
• https://gitlab.haskell.org/
• https://gitlab.freedesktop.org - this one has some interesting CI templates which could potentially be of use to us.
• https://gitlab.trisquel.org
• https://code.videolan.org

Conclusion

• gitlab.domain.tld seems to be the more prevalent option. Will go with that, i.e. gitlab.perlang.org in our case.

[perlun]

Steps remaining

• [x] Create virtual machine in an appropriate cloud provider
  • We have chosen Hetzner for this, since it's a provider that we already use elsewhere and are familiar with. Their pricing is also very reasonable and their web interface is also nice.
• [x] Create DNS record for gitlab.perlang.org
• [x] Install GitLab CE
• [ ] Set up OAuth-based authentication for the following authentication providers. This should lower the bar for people to contribute to the project.
  • [x] Google
  • [x] GitHub
  • [ ] GitLab (i.e. gitlab.com.
• [x] Create a perlang group

• [ ] Import the project from GitHub to perlang/perlang

  • I have done a test import. It took a while (perhaps around 45 minutes?) but did a fair job, including importing all the releases. What was not done correctly though was the PR/issue comments: references to #xxx in GitHub are inherently ambiguous in a GitLab context since it can be either !xxx or #xxx depending on whether the referred-to object is an MR or an issue. We would have to either go through all descriptions and comments and edit these manually (quite an effort, since there are a few thousand of them I think), or figure out an automated way for this. Perhaps it could be done using a GitLab Rails console snippet? I've used the GitLab Rails console previously and it feels like a reasonable fit for this.

  A "safer" approach would be to just use the Rails console and query all #xxx references that don't have a matching issue, and just list them. Editing them would be a manual process.

  • [x] A basic Rails snippet for going through all notes and fixing up MR references is now in place; I have executed it on the existing import at https://gitlab.perlang.org/perlang/perlang and it seems to work.
  • [ ] Submit the above as a feature request/bug report to GitLab, including the full Rails snippet.
  • [ ] Drop the project and re-run the import now that 0.5.0 has been released (so that the 0.5.0 release is also migrated to GitLab). This release is the final release being done on GitHub.
  • [ ] Release notes must be edited manually if we want them to refer to our GitLab URL:s (which we probably want).
  • Unsure if this is a "hard" requirement, but we could at least edit the 0.5.0 release notes to make mention about the new project home if we want to. OTOH, including it in the 0.5.1 release notes (which we intend to produce using GitLab CI) would perhaps be even more natural.
• [ ] Disable new group and project creation (to avoid spam because we are allowing anyone to signup to our GitLab instance)
• [ ] Ensure outgoing email/SMTP works, so that e.g. notifications can be sent.
  • Sent a support request to Hetzner about this, let's see what they say.
• [ ] Enable push mirroring from https://gitlab.perlang.org/perlang/perlang to https://github.com/perlang-org/perlang. This way, people can still continue using the GitHub repo to follow what's happening in the project to an extent.
• [ ] Mass-update all existing issues to:
  • [ ] Mark them as Locked, to prevent further comments to be posted on GitHub.
  • [ ] (Optional) As a courtesy to people reading the issues, add a link at the beginning of each issue (and preferably also PR) referring people to the new URL. (Is there a tool available for things like this? 🤔)
  • I think we might be able to avoid this, if we just create a single "pinned" issue in GitHub referring to the new GitLab URL. It will be one step of manual work less.

After the initial conversion has taken place

• [ ] Add CI configuration, so that we can run (some of the) CI on GitLab CI. GitHub CI jobs will still function, because of the push mirror mentioned above.

[perlun]

There we go, the import is now done so this will be the final comment written here. Let's continue in the new URL: https://gitlab.perlang.org/perlang/perlang/-/issues/423

(The work is not complete, but closing this so that we can keep all issues here on GitHub closed from now on)