Support `CI_JOB_TOKEN` auth #156

Open dosuken123 opened 4 years ago

GitLab CI injects CI_JOB_TOKEN to allow a pipeline job to access a project resource through public v4 API. Not all of APIs support job token auth, however, release creation API is already supported. So in theory, executing GITLAB_TOKEN=$CI_JOB_TOKEN should be suffice to run semantic-release.

However, as I tested https://gitlab.com/dosuken123/semantic-release-test/-/jobs/628942282, it didn't succeed as semantic-release tried remote Git repository authentication. I'm not sure why it's necessary as the tag creation is done via the above API call.

If CI_JOB_TOKEN auth is possible, it's very convenient to run semantic-release in GitLab CI as users don't need to create PAT.

For me, the CI_JOB_TOKEN works fine.

GitLab: 13.4.0-pre GitLab Runner: 13.3.1 (shared runner)

.gitlab-ci.yml

stages:
- release

release:
  needs: []
  image: registry.gitlab.com/rxap/gitlab-ci/semantic-release
  stage: release
  script: semantic-release
  variables:
    GITLAB_TOKEN: $CI_JOB_TOKEN
  interruptible: true
  rules:
  - if: '$CI_COMMIT_REF_NAME == "master"'
    when: on_success

@mmuenker I'm not able to reproduce a working release using your example with $CI_JOB_TOKEN, I get:

[9:15:28 PM] [semantic-release] › ✖  EGITNOPERMISSION Cannot push to the Git repository.

Can you think of anything different in your set up that would cause it to work?

is it possible that it was meant to work in gitlab 13.4.* and not 13.3 ?

I've tried this but I can't find any circumstances that made this work like @mmuenker says is possible. However, I also cannot find why it doesn't work - it seems like it fails before the gitlab plugin even does anything.

I have looked into this. I don't see how this could work at the moment, at least for /api/v4/.
What we need is an option to deactivate the testing of git pushes as CI tokens have not the permission to do these. See here.
The issue would need to be addressed in the sematic-release core. This is the line which results in the error. https://github.com/semantic-release/semantic-release/blob/master/index.js#L79

Could be that other rights are missing too.

I can confirm this has no chance to work as @secustor said. The error message is

'EGITNOPERMISSION',
  details: '**semantic-release** cannot push the version tag to the branch `semantic-release` on the remote Git repository with URL `blablabla`.\n' +

The use the CI_JOB_TOKEN would ease a lot the usage of semantic-release, but it looks complicated as long as the job token can't push tags (which is legit for security reasons)

It seems failing at verifyAuth in semantic-release (not semantic-release-gitlab module).

It's kinda make sense to auth if the token has a write permission to the repository, however, the release creation API technically doesn't require the write permission to create a git-tag in its subsequent process.

Alternatively, maybe Deploy Tokens should be able to be used instead. This is more appropriate than Personal Access Token or job token. However, there are no options to give write repository permission in the deploy token ATM, see https://gitlab.com/gitlab-org/gitlab/-/issues/23067.

Yes, the deploy and ci tokens are pretty limited (and for good reasons by default) The issue you reference have moved to this epic now: https://gitlab.com/groups/gitlab-org/-/epics/3559

Would you like to create a feature request on semantic-release to be able to have a verifyAuth(strict=false) or similar since indeed gitlab does not need write permissions to create the new tag?

This thread helped me better understand why the CI_JOB_TOKEN fails the verifyAuth step, even though the token does give permission to access GitLab's Tags API.

tl;dr: semantic-release creates tags directly through Git, not through the GitLab API, so the token needs to have repository write access (which CI_JOB_TOKEN does not have).

~@dosuken123 do you still have issues with this? I've just tried what @mmuenker showed as an example, and for me, it's working as it should. A new release is created in GitLab.~

By pure coincidence, I had a top level group variable that was a valid PAT. That's why it was working for me. Just confirmed that CI_JOB_TOKEN is not working as described by others.

Is someone actively working on it?

Ah cool. Then, we can close this issue :)

Sent from my iPhone

On Jan 25, 2022, at 7:41 PM, Aljoscha Pörtner @.***> wrote:

Is someone actively working on it?

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

I'd imagine gitlab-org/gitlab issue 223679 should resolve this but the timeline seems bit fluid (and far out there).

We've made it work with CI_JOB_TOKEN by making just a few small changes to semantic-release and semant-release/gitlab. As described in another thread here

Honestly, this is needed not just for convenience but also security. Semantic-release needs to have full write access to your code when there isn't any need or requirement for that. To use this tool you have to open a pretty large security hole that doesn't need to be opened.

I'd like to bring back an attention to that topic to use CI_JOB_TOKEN. Reference: https://github.com/semantic-release/semantic-release/issues/1729

Current limitation is identified, that semantic-release core is doing a check whether it can push a tag. On GitLab integration, creating a tag is not needed. Creation of a Release (via api, what this project is doing) is sufficient enough to make a release, because it results with a tag, as mentioned above (https://github.com/semantic-release/gitlab/issues/156#issuecomment-747163220). What is more, CI_JOB_TOKEN can create a release, while it cannot push a tag to satisfy the semantic-release core lifecycle.

For that purpose I think there should be possible to skip pushing a tags by semantic-release core and then this check could be optional.

The rationale behind that is:

CI_JOB_TOKEN are short-lived tokens with permissions derived from the triggerer. If ran by automation process they can access repo, but not modify the repo.
Personal Access Tokens are too wide to be used - the leakage or extracting from another peer poses the token's issuer into a risk to impersonate changes on his behalf.
Group level Access Tokens are not available in all organizations.
Project level Access Tokens are posed as similar leakage as Personal Access Tokens.

Having that said, CI_JOB_TOKEN is the most secure way of triggering a Release process on GitLab.

we hear the request. see https://github.com/semantic-release/semantic-release/issues/1729#issuecomment-1882385859 for our current response

since this depends on changes in core to enable this, having two threads related to this is making this conversation more difficult to track. closing this in favor of https://github.com/semantic-release/semantic-release/issues/1729. let's please consolidate further conversation there

https://gitlab.com/gitlab-org/gitlab/-/issues/468320 This might be possible soon in Gitlab 17.3 or 17.4

https://gitlab.com/gitlab-org/gitlab/-/issues/468320 This might be possible soon in Gitlab 17.3 or 17.4

Thanks for sharing @wwuck, I have not yet seen that!

With that functionality in place we would not need any semantic-release change. I'll therefore reopen this issue.

/cc @travi @JonasSchubert

I enabled the feature flag for the feature @wwuck mentioned, and it does pass: ✔ Allowed to push to the Git repository. However it now fails with the following error:

[7:31:15 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/gitlab"
[7:31:15 AM] [semantic-release] [@semantic-release/gitlab] › ℹ  Verify GitLab authentication (https://[GIT_URL]/api/v4)
[7:31:15 AM] [semantic-release] › ✘  Failed step "verifyConditions" of plugin "@semantic-release/gitlab"
[7:31:15 AM] [semantic-release] › ℹ  Start step "fail" of plugin "@semantic-release/gitlab"
[7:31:15 AM] [semantic-release] [@semantic-release/gitlab] › ℹ  Verify GitLab authentication (https://[GIT_URL]/api/v4)
[7:31:16 AM] [semantic-release] › ✘  Failed step "fail" of plugin "@semantic-release/gitlab"
[7:31:16 AM] [semantic-release] › ✘  EINVALIDGLTOKEN Invalid GitLab token.
The GitLab token (https://github.com/semantic-release/gitlab/blob/master/README.md#gitlab-authentication) configured in the GL_TOKEN or GITLAB_TOKEN environment variable must be a valid personal access token (https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) allowing to push to the repository [REPO].
Please make sure to set the GL_TOKEN or GITLAB_TOKEN environment variable in your CI with the exact value of the GitLab personal token.
[7:31:16 AM] [semantic-release] › ✘  EINVALIDGLTOKEN Invalid GitLab token.
The GitLab token (https://github.com/semantic-release/gitlab/blob/master/README.md#gitlab-authentication) configured in the GL_TOKEN or GITLAB_TOKEN environment variable must be a valid personal access token (https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) allowing to push to the repository [REPO].
Please make sure to set the GL_TOKEN or GITLAB_TOKEN environment variable in your CI with the exact value of the GitLab personal token.
AggregateError: 
    SemanticReleaseError: Invalid GitLab token.

From my inital research, it's because an api request is being made here: https://github.com/semantic-release/gitlab/blob/master/lib/verify.js#L67

Either the Job Token isn't allowed to make a request to the /projects endpoint at all (it's not listed here: https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html) or it's because of the use of the PRIVATE-TOKEN header, which should be JOB-TOKEN for Job Tokens.

I enabled the feature flag for the feature @wwuck mentioned, and it does pass: ✔ Allowed to push to the Git repository. However it now fails with the following error:

[7:31:15 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/gitlab"
[7:31:15 AM] [semantic-release] [@semantic-release/gitlab] › ℹ  Verify GitLab authentication (https://[GIT_URL]/api/v4)
[7:31:15 AM] [semantic-release] › ✘  Failed step "verifyConditions" of plugin "@semantic-release/gitlab"
[7:31:15 AM] [semantic-release] › ℹ  Start step "fail" of plugin "@semantic-release/gitlab"
[7:31:15 AM] [semantic-release] [@semantic-release/gitlab] › ℹ  Verify GitLab authentication (https://[GIT_URL]/api/v4)
[7:31:16 AM] [semantic-release] › ✘  Failed step "fail" of plugin "@semantic-release/gitlab"
[7:31:16 AM] [semantic-release] › ✘  EINVALIDGLTOKEN Invalid GitLab token.
The GitLab token (https://github.com/semantic-release/gitlab/blob/master/README.md#gitlab-authentication) configured in the GL_TOKEN or GITLAB_TOKEN environment variable must be a valid personal access token (https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) allowing to push to the repository [REPO].
Please make sure to set the GL_TOKEN or GITLAB_TOKEN environment variable in your CI with the exact value of the GitLab personal token.
[7:31:16 AM] [semantic-release] › ✘  EINVALIDGLTOKEN Invalid GitLab token.
The GitLab token (https://github.com/semantic-release/gitlab/blob/master/README.md#gitlab-authentication) configured in the GL_TOKEN or GITLAB_TOKEN environment variable must be a valid personal access token (https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) allowing to push to the repository [REPO].
Please make sure to set the GL_TOKEN or GITLAB_TOKEN environment variable in your CI with the exact value of the GitLab personal token.
AggregateError: 
    SemanticReleaseError: Invalid GitLab token.

From my inital research, it's because an api request is being made here: https://github.com/semantic-release/gitlab/blob/master/lib/verify.js#L67

Either the Job Token isn't allowed to make a request to the /projects endpoint at all (it's not listed here: https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html) or it's because of the use of the PRIVATE-TOKEN header, which should be JOB-TOKEN for Job Tokens.

Not sure, but I think header Authorization: Bearer <token> can handle both? (instead PRIVATE-TOKEN or JOB-TOKEN)

~~EDIT: it does not~~

EDIT EDIT: it does (EXAMPLE: https://docs.gitlab.com/ee/api/jobs.html#get-job-tokens-job)! But CI_JOB_TOKEN has no access to projects api https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html

Judging from the age and last update on this issue, the job token probably won't ever be able to access the projects endpoint, atleast not in the near future.

@fgreinacher Should we just skip the permissions check in https://github.com/semantic-release/gitlab/blob/master/lib/verify.js if GITLAB_TOKEN is equal to the CI_JOB_TOKEN?

Hm, how would we know that the token is a job token?

Hm, how would we know that the token is a job token?

We'd require the GITLAB_TOKEN to be set to the CI_JOB_TOKEN in the .gitlab-ci.yml like so:

variables:
  GITLAB_TOKEN: $CI_JOB_TOKEN

and inside of the verify.js we'd get both tokens and check if they are equal.

Hm, how would we know that the token is a job token?

We'd require the GITLAB_TOKEN to be set to the CI_JOB_TOKEN in the .gitlab-ci.yml like so:
variables:
  GITLAB_TOKEN: $CI_JOB_TOKEN
and inside of the verify.js we'd get both tokens and check if they are equal.

Is there anything against additionally defaulting the GL_TOKEN to the CI_JOB_TOKEN?

I don't think we should default the GitLab token to CI_JOB_TOKEN as long as it's not enough to support all use cases. It would be super confusing for users when some things work and some don't.

One issue both for and against using the JOB TOKEN is that it is associated with the user that triggered the pipeline. This improves audit trails as releases and child pipelines now show the actual user. The other aspect is of course, that pipelines will play differently depending on the users role, which some might find unexpected, and others will find desirable.

I am in the desirable camp: While it may cause unexpected variation, using CI_JOB_TOKEN prevents privilege escalation via semantic-release.

semantic-release / gitlab

Support `CI_JOB_TOKEN` auth #156