Deploying docs on GitLab CI with version-selection feature #2061

Open tamasgal opened 1 year ago

I am trying to get Documenter.jl to work on our self-hosted GitLab instance but I failed to set it up so that also versions of the docs are kept and commited to versions.js to the gh-pages branch.

A couple of intrinsic problems:

GitLab pages is fundamentally different compared to GitHub. GitLab requires a public/ folder to be declared as artifact inside a job named pages and that folder will be stored and exposed when visiting the pages URL
GitHub holds all the data in a separate branch called gh-pages and uses that as the data source for the pages
The way the deployment works (using an SSH key-pair) is problematic in GitLab since there is no way to create a "project-specific SSH-key authentication", which means that one of the users needs to upload a pub-key to their own GitLab account and add the private-key as a CI variable named DOCUMENTER_KEY. This means that everyone who has access to the CI settings can potentially impersonate the user

What works:

Basically not much, or not enough ;)
Documenter.jl is able to create the documentation and by creating a folder called public/, moving the build/* files to that folder and declare the public/ folder as a job artifact of the job named pages, the documentation is deployed correctly
Documenter.jl also creates the gh-pages branch and updates it also each time the CI job is executed

What does not work:

The git command is not available in the official Julia Docker images (I used our own Julia image which has Git installed) and it is required (switching to libgit would be better, as mentioned in another issue)
versions.jsonly contains an empty list var DOC_VERSIONS = []; and also only the dev folder instead of (git) tagged versions which are named accordingly vX.Y.Z
There is no mechanic which deploys the contents of the gh-pages branch to the actual pages, since all the gh-branch stuff is happening inside the Documenter.jl process which uses a temporary folder to do the Git commit and push

How to proceed

I am not really familiar with Documenter.jl but I think that the fix is relatively easy, given that all of the functionality to create the HTML/JS stuff should technically work in any Git instance. I think we can boil this down to two action points:

[ ] Keep the contents of the gh-pages branch when creating the new commit inside the pages job and move their contents to public/ so that it can be picked up by GitLab correctly
[ ] Fix the problem that versions.js is not populated correctly

The CI job would then look like this:

pages:
  image: julia-image-which-contains-git:X.Y
  stage: deploy
  script:
    - |
      julia --project=docs -e '
        using Pkg
        Pkg.develop(PackageSpec(path=pwd()))
        Pkg.instantiate()
        using Documenter: doctest
        using PackageName
        doctest(PackageName)
        include("docs/make.jl")'
    - mkdir -p public
    - mv docs/build/* public/
  artifacts:
    paths:
      - public
  only:
    - main

Questions

By any chance, does anybody got this working somehow?
Any ideas why versions.js is not populated?

I will try to spend time on this (I have to, since I need this ;) but I am a total noob regarding the Documenter.jl internals so I am happy when someone gives me a headstart.

Could you not deploy to gh-pages, and then have another workflow there that triggers on push and copies the whole branch to public/?

Yes, that could work but the CI needs a .gitlab-ci.yml inside that branch, since CI pipelines needs to be defined per branch via the file, which also means that Documenter.jl would be responsible to create the job definition for the pages.

The other problem of course is also that versions.js is not populated.

which also means that Documenter.jl would be responsible to create the job definition for the pages.

Documenter doesn't overwrite or delete existing stuff generally. So you should be able to add it to the branch yourself and it should stay there.

The other problem of course is also that versions.js is not populated.

Hard to say anything without an MWE. But the versions.js is just generated based on the directories in the checked out (and updated) gh-pages branch. If the versioned directories are there, then they should end up in versions.js.

Here is the project I am working on: https://git.km3net.de/tgal/JuliaGitLabPages.jl

Yes, I am aware that I can add the file manually (and we all know what this means: two months later we scratch our heads and wondering why it's not working for the next project), which I will also try immediately. I was currently trying to understand where exactly I could automate this in Documenter.jl. I guess somewhere here: https://github.com/JuliaDocs/Documenter.jl/blob/99a541c39175b9f2ce8c9d5016d2a7971d6712b2/src/deploydocs.jl#L327 😉

So this is the pipeline of a commit which I also tagged with a new version: https://git.km3net.de/tgal/JuliaGitLabPages.jl/-/pipelines/37863

The existing tags are listed in the main branch (https://git.km3net.de/tgal/JuliaGitLabPages.jl/-/jobs/207498#L85) but right above you can see that Documenter.jl does not care about versions.

OK, I added the following CI script to gh-pages:

pages:
  stage: deploy
  script:
    - mkdir .public
    - mv * .public/
    - mv .public public
  artifacts:
    paths:
      - public

and then modified the version in Project.toml from 0.4.0 to 1.0.0 and also edited the docs/src/index.md just to have a another text to display for 1.0.0.

░ tamasgal@silentbox:JuliaGitLabPages  main
░ 11:18:44 > vim Project.toml

░ tamasgal@silentbox:JuliaGitLabPages  main ● took 5s
░ 11:18:51 > vim docs/src/index.md

░ tamasgal@silentbox:JuliaGitLabPages  main ● took 5s
░ 11:19:00 > git add docs/src/index.md

░ tamasgal@silentbox:JuliaGitLabPages  main ●●
░ 11:19:03 > git add Project.toml

░ tamasgal@silentbox:JuliaGitLabPages  main ●
░ 11:19:05 > git commit -m 'Bump version to 1.0.0'
[main 2a90925] Bump version to 1.0.0
 2 files changed, 2 insertions(+), 2 deletions(-)

░ tamasgal@silentbox:JuliaGitLabPages  main ⇡
░ 11:19:13 > git tag -a v1.0.0 -m 'Release 1.0.0'

░ tamasgal@silentbox:JuliaGitLabPages  main ⇡
░ 11:19:24 > git push
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (7/7), 1015 bytes | 1015.00 KiB/s, done.
Total 7 (delta 4), reused 0 (delta 0), pack-reused 0
To git.km3net.de:tgal/JuliaGitLabPages.jl.git
   4b13a0f..2a90925  main -> main
 * [new tag]         v1.0.0 -> v1.0.0

Here is the relevant pipeline output (https://git.km3net.de/tgal/JuliaGitLabPages.jl/-/jobs/207517) and I just saw that Commit tag: "" is showing an empty tag, which is probably related to a wrong CI environment variable mapping. I am a bit confused since

https://github.com/JuliaDocs/Documenter.jl/blob/99a541c39175b9f2ce8c9d5016d2a7971d6712b2/src/deployconfig.jl#L592

shows that it's taken from CI_COMMIT_TAG which should be there.

│ GitLab config: 74│ Commit branch: "main" 75│ Pull request IID: "" 76│ Repo slug: "tgal-juliagitlabpages-jl" 77│ Commit tag: "" 78│ Pipeline source: "push" 79│ Detected build type: devbranch 80│ - ✔ ENV["CI_COMMIT_BRANCH"] matches devbranch="main" 81│ - ✔ ENV["DOCUMENTER_KEY"] exists and is non-empty 82└ Deploying to folder "dev": ✔ 83Initialized empty Git repository in /tmp/jl_TGmGdL/.git/ 84Failed to add the host to the list of known hosts (/root/.ssh/known_hosts). 85From git.km3net.de:tgal/JuliaGitLabPages.jl 86 [new branch] gh-pages -> upstream/gh-pages 87 [new branch] main -> upstream/main 88 [new tag] v1.0.0 -> v1.0.0 89 [new tag] v0.1.0 -> v0.1.0 90 [new tag] v0.2.0 -> v0.2.0 91 [new tag] v0.3.0 -> v0.3.0 92 * [new tag] v0.4.0 -> v0.4.0 93Branch 'gh-pages' set up to track remote branch 'gh-pages' from 'upstream'. 94Switched to a new branch 'gh-pages' 95rm 'dev/assets/documenter.js' 96rm 'dev/assets/search.js' 97rm 'dev/assets/themes/documenter-dark.css' 98rm 'dev/assets/themes/documenter-light.css' 99rm 'dev/assets/themeswap.js' 100rm 'dev/assets/warner.js' 101rm 'dev/index.html' 102rm 'dev/search/index.html' 103rm 'dev/search_index.js'

I made some progress...

Apparently adding tags to the only: section in the .gitlab-ci.yml is mandatory, otherwise the CI_COMMIT_TAG is not populated. A CI job is ran for both the commit on e.g. main and then the tag.
DOCUMENTER_KEY is empty in the job which is executed for the tag and I don't understand why.

Here is the .gitlab-ci.yml I am using right now:

docs:
  image: docker.km3net.de/base/julia:1.6
  stage: deploy
  script:
    - |
      julia --project=docs -e '
        using Pkg
        Pkg.develop(PackageSpec(path=pwd()))
        Pkg.instantiate()
        using Documenter: doctest
        using JuliaGitLabPages
        doctest(JuliaGitLabPages)
        include("docs/make.jl")'
  only:
    - main
    - tags

And the pipeline output which reveals that DOCUMENTER_KEY is an empty string:

│ GitLab config: 74│ Commit branch: "" 75│ Pull request IID: "" 76│ Repo slug: "tgal-juliagitlabpages-jl" 77│ Commit tag: "v1.1.0" 78│ Pipeline source: "push" 79│ Detected build type: release 80│ - ✔ ENV["CI_COMMIT_TAG"] contains a valid VersionNumber 81│ - ✘ ENV["DOCUMENTER_KEY"] exists and is non-empty 82└ Deploying to folder "v1.1.0": ✘

I added the CI variable as protected, but according to the docs it also applies to tags, so one needs to add a wildcard to the tags (e.g. v*):

OK (sorry for the spam), I got it working ;)

Thanks for the input so far. Now the question is: could we add that .gitlab-ci.yml to the set of files which are copied to the pages branch? I'd open a PR and also add a section to the docs.

https://tgal.pages.km3net.de/JuliaGitLabPages.jl/stable/

I don't have any access to any of the repos/logs/pipelines etc.

However, any docs improvements are always welcome. I think the only docs about GitLab we have are here: https://documenter.juliadocs.org/stable/man/hosting/#Documenter.GitLab We could probably turn it into a whole section -- no need to jam everything into the docstring.

.gitlab-ci.yml to the set of files which are copied to the pages branch?

I don't want to add that in an ad-hoc way. But we could perhaps define a function that each DeployConfig can extend that then does some post-processing on the copied files, and it could be copied there for GitLab. Probably should be opt-in though, since it might clash with existing uses of the GitLab deploy config?

Ah sorry, I did not realise it was "private", it's now public 😉

Yes, I thought about creating a dedicated section.

I am not really familiar with the underlying mechanics of Documenter.jl yet but since GitLab only allows to deploy a single instance of pages, it makes sense to automatically add the .gitlab-ci.yml to the gh-pages branch. The only way it would clash is if the user already has another pages-job on another branch, but that's probably a very rare case.

The other approach would be -- as mentioned before -- copying the whole gh-pages content to the public/ and then the doc generation job itself needs to be called pages and provide the public-artifact. The only reason it would make sense is that the user can easily generate additional files in the same job and publish them "manually" by adjusting the pages job to copy/alter files to the public/ folder. I am pretty sure that it's a very rare case...

The only way it would clash is if the user already has another pages-job on another branch, but that's probably a very rare case.

But there are existing users of the GitLab deployconfig, and they must have some setup that this may clash with. Also, as this would add a CI workflow that may accrue cost or may have some invalid configuration (e.g. something like it not using self-hosted runners), I don't think this should be hoisted upon the user without their explicit decision. So a well-documented option in GitLab(...) still seems reasonable to me.

The other approach would be -- as mentioned before -- copying the whole gh-pages content to the public/ and then the doc generation job itself needs to be called pages and provide the public-artifact.

First, I think this would be more complex in Documenter, since it would require copying out stuff from the deploydocs' temporary gh-pages Git clone. But it would also mean that any changes to gh-pages that happen outside of the Documenter workflow do not get deployed (until the Documenter workflow runs again anyway). And finally, a separate pages workflow that lives on gh-pages branch also aligns with GitHub better (technically, every time you push to gh-pages, GitHub runs an Action, which you can actually override nowadays).

So my feeling is that a separate workflow for gh-pages -> pages copy is better.

Yes, I fully agree with not breaking existing workflows!

I also agree that the gh-pages approach is the better one. It works very well for me now with the setup I have.

...and I also found that there is indeed a possibility to add project-specific SSH authentication via so-called "Deploy keys": https://docs.gitlab.com/ee/user/project/deploy_keys/index.html

I'd really like to help with the implementation but I am not fully into the architecture of Documenter.jl yet, so let me throw in something to begin with 😉

What about changing the default branch in deployconfig() for GitLab to gl-pages so that it does not collide with GitHub deployments?
We can add an option which enables/disables the automatic addition of .gitlab-ci.yml to the branch.

What about changing the default branch in deployconfig() for GitLab to gl-pages so that it does not collide with GitHub deployments?

In principle, this would be sensible. However, (1) changing a default is a breaking change, and (2) the branch argument currently doesn't come from DeployConfig, but is separate from that logic. I am not sure it's worth going through the hassle of changing all that for this. Maybe just keep it as a documentation-level recommendation to change branch = "gl-pages" for GitLab deployments?

OK I am fine with that, I'll then add a PR with a dedicated GitLab docs section ;)

I have recently been dealing with automating Documenter on a self-hosted GitLab instance and I have to say this discussion was quite helpful.

I just wanted to add that it is also possible with very small modification to make Documenter also work with access tokens rather than SSH keys. On my case, I have a Gitlab group that hosts all the julia packages and created a group access token with write_repository and copied that as a CI/CD variable for the group pipelines/jobs.

To make Documenter use a token instead of the SSH key you need to create a custom DeployConfig which is for the most part equivalent to the standard GitLab one, but you have to specify HTTPS as authentication method and provide the authentication URL.

In my case I just created a stuct accordingly with the following code:

@kwdef struct GitLabHTTPS <: DeployConfig
    commit_branch::String = get(ENV, "CI_COMMIT_BRANCH", "")
    pull_request_iid::String = get(ENV, "CI_EXTERNAL_PULL_REQUEST_IID", "")
    repo_path::String = get(ENV, "CI_PROJECT_PATH", "")
    commit_tag::String = get(ENV, "CI_COMMIT_TAG", "")
    pipeline_source::String = get(ENV, "CI_PIPELINE_SOURCE", "")
end

Documenter.authentication_method(::GitLabHTTPS) = HTTPS

function Documenter.authenticated_repo_url(cfg::GitLabHTTPS) 
    token = get(ENV,"GROUP_WRITE_REPOSITORY_TOKEN","")
    host = get(ENV,"CI_SERVER_HOST", "")
    return "https://documenter-ci:$token@$host/$(cfg.repo_path).git"
end

The Documenter.deploy_folder method is basicaly a copy of https://github.com/JuliaDocs/Documenter.jl/blob/67a33c08a00208ba36dd858bba957a02680d5cb9/src/deployconfig.jl#L597-L694 with small changes to check for the token variable (which in my case I save inside GROUP_WRITE_REPOSITORY_TOKEN) rather than checking for DOCUMENTER_KEY.

If this is of interest I might try to submit a PR for adding HTTPS as possible authentication method to the current GitLab deploy config struct, similarly to how it's done for GitHub

@disberd Is this working well for you? Any chance that you might put together a PR with GitLab pages support or have you posted a detailed write-up of the full working example anywhere?

@ianfiske yes we have it running for automatic documentation generation since I posted and it works quite well.

I'll try to find time in the next weeks for either a PR or a detailed example

JuliaDocs / Documenter.jl