Support for on-prem GitLab

[muffato]

576 adds support for remote registries hosted at gitlab.com. GitLab can also be deployed on prem, i.e. at a different URL.

For this to work, the code would have to:

1. have an option to configure Pages (which wouldn't be at gitlab.io)
2. build the path to the raw container.yaml from the actual server rather than hardcoding gitlab.com (in shpc docgen)

For pages, I propose to extend shpc/settings.yml to allow a mapping between the server of the git repository and the location of its pages. The default config would be:

pages_url:
  github.com: github.io
  gitlab.com: gitlab.io

and users could add there their own settings.

This would also remove the need for defining classes GitHub and GitLab in shpc/main/registry/remote.py (and use those in shpc/main/registry/__init__.py). A registry would be considered as remote as soon as it starts with https://, and to be valid, its server would have to be found in the config file.

[georgiastuart]

I need this feature! Has any work towards it been done? Happy to take a crack at it if it's up for grabs.

[vsoch]

I think maybe we added this - @muffato was there more work to do or did we forget to close it?

[muffato]

Hi @georgiastuart

Thanks to @phue (#631), there is now support on the main branch for cases when the Pages are deployed on a sub-domain of the self-hosted GitLab itself, cf https://github.com/singularityhub/singularity-hpc/issues/630#issue-1584422695 and https://github.com/singularityhub/singularity-hpc/issues/630#issuecomment-1430456143

However that pattern seems to contravene GitLab's rule that Pages cannot be on a subdomain of GitLab itself, see https://docs.gitlab.com/ee/administration/pages/#prerequisites @phue : how did you manage to configure Pages that way ? For instance, here at the Sanger institute, we use gitlab.internal.sanger.ac.uk and pages.internal.sanger.ac.uk, so it still wouldn't work.

With @vsoch we started some work in #585 to address both this issue and #581 . The pull-request now has a few conflicts that could probably be solved, but I don't think it's complete. I somehow can't see how it was supposed to handle self-hosted GitLab. At the moment, the part that works the best in #585 is the support for private registries that don't expose the JSON over Pages, and for which shpc falls back to git clone.

In summary, @georgiastuart , can you first check your Pages setup and if it works with the code on main ? If not, let's solve that first, and we'll see what to do with the rest of #585 after

[phue]

@phue : how did you manage to configure Pages that way ? For instance, here at the Sanger institute, we use gitlab.internal.sanger.ac.uk and pages.internal.sanger.ac.uk, so it still wouldn't work.

Hm I see, looks like there are many ways to deploy gitlab. I did not configure gitlab myself, but I have worked at two institutions where it was configured like this ( e.g gitlab.rlp.net and pages.gitlab.rlp.net), so I (mistakenly) assumed it to be a common deployment scenario. To handle such different scenarios, https://github.com/singularityhub/singularity-hpc/pull/631 isn't very helpful indeed but would require a config option for pages_url as you suggested above.

[georgiastuart]

I'm not ready to PR yet, but I got this working by implementing a "generic" provider that just checks if the library.json file exists.

Our gitlab pages domain is completely separate from our gitlab domain.

[georgiastuart]

Partly I'm not ready to PR since I had to jam in some hacky stuff to make it work with a URL that has a port for the container registry in the container.yaml files, haha.