Failed to backmerge branches.: xJ: Command failed with ENOENT: git fetch

syphernl commented 1 day ago

We are running in to a rather odd issue with this plugin. In one particular project the backmerges always fail with:

Failed to backmerge branches.

xJ: Command failed with ENOENT: git fetch 'https://gitlab-ci-token:[secure]@gitlab.com/REDACTED' spawnSync git ENOENT

We have GITLAB_TOKEN set as Masked+Hidden variable. The token itself is fine, since semantic-release/gitlab can create releases + tags just fine using it. Git is installed in the container as well:

~ # git --version
git version 2.45.2

Relevant part of .releaserc.js:

    [
      "@kilianpaquier/semantic-release-backmerge",
      {
        targets: [
          { from: "production", to: "develop" },
          { from: "acceptance", to: "develop" }
        ],
        title: "ci(release): fix mergeback from ${ from } into ${ to } [skip ci]",
      }
    ],

Logs:

[8:16:48 AM] [semantic-release] › ℹ  Start step "success" of plugin "@kilianpaquier/semantic-release-backmerge"
[8:16:48 AM] [semantic-release] [@kilianpaquier/semantic-release-backmerge] › ℹ  Current branch 'acceptance' matches following configured backmerge targets: '[{"from":"acceptance","to":"develop"}]'. Performing backmerge.
[8:16:48 AM] [semantic-release] › ✘  Failed step "success" of plugin "@kilianpaquier/semantic-release-backmerge"
[8:16:48 AM] [semantic-release] › ℹ  Start step "fail" of plugin "@semantic-release/gitlab"
[8:16:49 AM] [semantic-release] [@semantic-release/gitlab] › ℹ  Commented on issue REDACTED: https://gitlab.com/REDACTED/-/issues/X.
[8:16:49 AM] [semantic-release] › ✔  Completed step "fail" of plugin "@semantic-release/gitlab"
[8:16:49 AM] [semantic-release] › ℹ  Start step "fail" of plugin "semantic-release-slack-bot"
[8:16:49 AM] [semantic-release] [semantic-release-slack-bot] › ℹ  Notifying on fail skipped
[8:16:49 AM] [semantic-release] › ✔  Completed step "fail" of plugin "semantic-release-slack-bot"
[8:16:49 AM] [semantic-release] › ✘  EBACKMERGE Failed to backmerge branches.
Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'marked-terminal' imported from /usr/local/lib/node_modules/semantic-release/index.js
    at packageResolve (node:internal/modules/esm/resolve:854:9)
    at moduleResolve (node:internal/modules/esm/resolve:927:18)
    at defaultResolve (node:internal/modules/esm/resolve:1169:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:542:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:510:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:239:38)
    at ModuleLoader.import (node:internal/modules/esm/loader:472:34)
    at defaultImportModuleDynamicallyForModule (node:internal/modules/esm/utils:214:31)
    at importModuleDynamicallyCallback (node:internal/modules/esm/utils:253:12)
    at terminalOutput (file:///usr/local/lib/node_modules/semantic-release/index.js:29:43) {
  code: 'ERR_MODULE_NOT_FOUND'
}

I have tried to reproduce the issue in a separate repository, but for the backmerges seem to work fine there. They both use the same config in .releaserc.js. Is there a way to get more logging out of this plugin to see what it is actually doing and why it is failing?

kilianpaquier commented 1 day ago

Hello, thank you for opening this issue !

You can enable semantic-release in debug mode (--debug) but I don't think I added debug logs.

Some questions :

What's the semantic-release version you're using ?
The first provided error isn't in the logs at the end of your issue, are they related ?
What OS is being used for this job ?

syphernl commented 1 day ago

Hi @kilianpaquier, thanks for your quick reply :)

What's the semantic-release version you're using ?

We run version 24.1.2.

The first provided error isn't in the logs at the end of your issue, are they related ?

The first error (and the title) is the error that showed up in the "Release is failing" issue created by Semantic-Release.

What OS is being used for this job ?

We run this job in a container based on Alpine 3.20.3.

It is rather strange that it is failing with one project, but not with the other. I have doublechecked and compared GitLab settings, pipeline configs, access token settings etc and there aren't any differences as far as I can see. The ENOENT seems to indicate it cannot run git (because it does not exist) but it is part of the container and works fine in another project..

kilianpaquier commented 1 day ago

Thank you for this reply.

Could you try installing marked-terminal manually during the CI with npm i -g marked-terminal and rerun the thing ?

You can run in --dry-run since it'll at least fetch and run a git push --dry-run. If the issue with git is still there it'll fail.

For the dry run to go through the whole process, you'll at least need to have a commit eligible to releasing 🙂.

syphernl commented 1 day ago

In the before_script I added the install command, but the error with marked-terminal is still the same.

Could this ENOENT problem be permission related? If it is using the output from git ls-remote --heads it will get an URL which Gitlab CI used which includes the (short-lived and limited) CI_JOB_TOKEN that does not have write permissions, while the GITLAB_TOKEN is expected to have that. Still weird, since the error it claims to be failing on is git fetch which is allowed via the CI_JOB_TOKEN..

Running with env DEBUG=semantic-release:* didn't show any info for the plugin (which was expected as you indicated earlier) but I did see this:

2024-10-21T11:46:22.017Z semantic-release:get-git-auth-url Verifying ssh auth by attempting to push to  https://gitlab-ci-token:[secure]@gitlab.com/REDACTED.git
2024-10-21T11:46:22.510Z semantic-release:git ExecaError: Command failed with exit code 128: git push --dry-run --no-verify 'https://gitlab-ci-token:[secure]@gitlab.com/REDACTED.git' 'HEAD:acceptance'
remote: You are not allowed to push code to this project.
fatal: unable to access 'https://gitlab.com/REDACTED.git/': The requested URL returned error: 403
    at getFinalError (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/return/final-error.js:6:9)
    at makeError (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/return/result.js:108:16)
    at getAsyncResult (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/methods/main-async.js:167:4)
    at handlePromise (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/methods/main-async.js:150:17)
    at async verifyAuth (file:///usr/local/lib/node_modules/semantic-release/lib/git.js:206:5)
    at async default (file:///usr/local/lib/node_modules/semantic-release/lib/get-git-auth-url.js:93:5)
    at async run (file:///usr/local/lib/node_modules/semantic-release/index.js:67:27)
    at async Module.default (file:///usr/local/lib/node_modules/semantic-release/index.js:278:22)
    at async default (file:///usr/local/lib/node_modules/semantic-release/cli.js:55:5)
2024-10-21T11:46:22.511Z semantic-release:get-git-auth-url SSH key auth failed, falling back to https.
2024-10-21T11:46:30.600Z semantic-release:get-tags found tags for branch review: []
2024-10-21T11:46:31.097Z semantic-release:get-tags found tags for branch acceptance: [ <list_of_tags_here> ]
2024-10-21T11:46:31.490Z semantic-release:get-tags found tags for branch production: [ <list_of_tags_here> ]
[11:46:31 AM] [semantic-release] › ✔  Run automated release from branch acceptance on repository https://gitlab-ci-token:[secure]@gitlab.com/REDACTED.git
[11:46:32 AM] [semantic-release] › ✔  Allowed to push to the Git repository

syphernl commented 1 day ago

If I do a git fetch myself with CI_JOB_TOKEN or GITLAB_TOKEN in the before_script it just works:

$ git fetch "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/REDACTED.git"
From https://gitlab.com/REDACTED
 * branch            HEAD       -> FETCH_HEAD

$ git fetch "https://gitlab-ci-token:${GITLAB_TOKEN}@gitlab.com/REDACTED.git"
From https://gitlab.com/REDACTED
 * branch            HEAD       -> FETCH_HEAD

Pushing also works fine, assuming the GITLAB_TOKEN is used for that:

$ git push --dry-run --no-verify "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/REDACTED.git" 'HEAD:acceptance' || echo "Expected to fail"
remote: You are not allowed to push code to this project.
fatal: unable to access 'https://gitlab.com/REDACTED.git/': The requested URL returned error: 403
Expected to fail

$ git push --dry-run --no-verify "https://gitlab-ci-token:${GITLAB_TOKEN}@gitlab.com/REDACTED.git" 'HEAD:acceptance'
Everything up-to-date

kilianpaquier commented 1 day ago

Thank you for this investigation.

You were saying that for another project it was working fine ? Can you confirm that's the same CI configuration and that the GITLAB_TOKEN has the appropriate rights for both projects if shared or the appropriate rights for each projects if not shared ?

You were saying that git was installed on your runner, are you using the runner in shell mode or in docker mode ? If in docker mode, did you ensure that git is installed in the docker image either by default or during the CI ?

Are you using CI_JOB_TOKEN or a custom generated GITLAB_TOKEN ?

Could you provide which rights you gave to the GITLAB_TOKEN ? @semantic-release/gitlab uses only the API to create the release. However it's semantic-release itself which creates the tag and I don't really know if it uses the GITLAB_TOKEN or pushes with something else.

syphernl commented 1 day ago

Thank you for this investigation.

You were saying that for another project it was working fine ?

Yes, a standalone project (very barebones) works fine. The original project and a fork of the project are experiencing these issues with backmerges. But the pipeline config and branch/tag/variables are configured the same for each of them.

Can you confirm that's the same CI configuration and that the GITLAB_TOKEN has the appropriate rights for both projects if shared or the appropriate rights for each projects if not shared ?

Yes, the token has appropriate scopes. All 3 projects (original, work, barebones test) have their own tokens with the same permissions.

You were saying that git was installed on your runner, are you using the runner in shell mode or in docker mode ? If in docker mode, did you ensure that git is installed in the docker image either by default or during the CI ?

We have a custom image with semantic-release and git included.

Are you using CI_JOB_TOKEN or a custom generated GITLAB_TOKEN ?

The plugin and semantic release both require/use the GITLAB_TOKEN. Since the CI_JOB_TOKEN gets masked I cannot verify what is in the [secret] portion of the URL.

I have tried to update the git remote prior to running semantic release, to incorporate the GITLAB_TOKEN in the URL but it makes no difference.

kilianpaquier commented 1 day ago

If not too confidential, could you provide here the semantic-release job configuration in your .gitlab-ci.yml please ?

kilianpaquier commented 1 day ago

Hi so I tried to reproduce the issue here https://gitlab.com/kilianpaquier/semantic-release-testing without success:

with a GITLAB_TOKEN, masked, hidden and non-protected, with Developer role and only api scope, backmerge works on a non-protected branch (staging to develop)
with a GITLAB_TOKEN, masked and hidden, with Maintainer role and all api, read_repository and write_repository, backmerge works on a protected branch (main to develop)

From this quote, it seems @semantic-release/git is also impacted like the provided token doesn't have enough rights ...

In the before_script I added the install command, but the error with marked-terminal is still the same.

Could this ENOENT problem be permission related? If it is using the output from git ls-remote --heads it will get an URL which Gitlab CI used which includes the (short-lived and limited) CI_JOB_TOKEN that does not have write permissions, while the GITLAB_TOKEN is expected to have that. Still weird, since the error it claims to be failing on is git fetch which is allowed via the CI_JOB_TOKEN..

Running with env DEBUG=semantic-release:* didn't show any info for the plugin (which was expected as you indicated earlier) but I did see this:

2024-10-21T11:46:22.017Z semantic-release:get-git-auth-url Verifying ssh auth by attempting to push to  https://gitlab-ci-token:[secure]@gitlab.com/REDACTED.git
2024-10-21T11:46:22.510Z semantic-release:git ExecaError: Command failed with exit code 128: git push --dry-run --no-verify 'https://gitlab-ci-token:[secure]@gitlab.com/REDACTED.git' 'HEAD:acceptance'
remote: You are not allowed to push code to this project.
fatal: unable to access 'https://gitlab.com/REDACTED.git/': The requested URL returned error: 403
    at getFinalError (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/return/final-error.js:6:9)
    at makeError (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/return/result.js:108:16)
    at getAsyncResult (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/methods/main-async.js:167:4)
    at handlePromise (file:///usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/methods/main-async.js:150:17)
    at async verifyAuth (file:///usr/local/lib/node_modules/semantic-release/lib/git.js:206:5)
    at async default (file:///usr/local/lib/node_modules/semantic-release/lib/get-git-auth-url.js:93:5)
    at async run (file:///usr/local/lib/node_modules/semantic-release/index.js:67:27)
    at async Module.default (file:///usr/local/lib/node_modules/semantic-release/index.js:278:22)
    at async default (file:///usr/local/lib/node_modules/semantic-release/cli.js:55:5)
2024-10-21T11:46:22.511Z semantic-release:get-git-auth-url SSH key auth failed, falling back to https.
2024-10-21T11:46:30.600Z semantic-release:get-tags found tags for branch review: []
2024-10-21T11:46:31.097Z semantic-release:get-tags found tags for branch acceptance: [ <list_of_tags_here> ]
2024-10-21T11:46:31.490Z semantic-release:get-tags found tags for branch production: [ <list_of_tags_here> ]
[11:46:31 AM] [semantic-release] › ✔  Run automated release from branch acceptance on repository https://gitlab-ci-token:[secure]@gitlab.com/REDACTED.git
[11:46:32 AM] [semantic-release] › ✔  Allowed to push to the Git repository

From our discussion, you are talking about CI_JOB_TOKEN too, backmerge plugin doesn't read it in its code, in which way are you using this token ?

syphernl commented 1 day ago

Sure, here it is:

.release/semantic-release:
  stage: release
  allow_failure: false
  image:
    name: ${CI_REGISTRY}/myorg/containers/semantic-release
    entrypoint: ['']
  script:
    # Ensure the git directory is marked as safe, otherwise we cannot run
    - git config --global --add safe.directory "${CI_PROJECT_DIR}"

    # Fetch all tags
    - git fetch --tags

    # Check if dry run is enabled, otherwise append it to the options
    - if [[ "${SEMREL_DRY_RUN}" == "true" ]]; then dry_run_opt="--dry-run"; fi

    # Run Semantic-release
    - semantic-release --repository-url="${CI_REPOSITORY_URL}" ${dry_run_opt} | tee semantic-output.txt

    - |
      # Export the version as artifact
      RELEASE_VERSION=$(sed -n 's/.*Published release \([^ ]*\).*/\1/p' semantic-output.txt)
      echo "Released: ${RELEASE_VERSION}"
      echo "RELEASE_VERSION=${RELEASE_VERSION}" > .release-version.env
  artifacts:
    expire_in: 1 hour # Only needs to exist briefly until GL backend has processed it
    reports:
      dotenv: .release-version.env

release/acceptance:
   extends: .release/semantic-release
   rules:
       - if: $CI_COMMIT_BRANCH == "acceptance"

kilianpaquier commented 1 day ago

Thanks ! Could you try to run with the following change:

change semantic-release --repository-url="${CI_REPOSITORY_URL}" ${dry_run_opt} | tee semantic-output.txt
to semantic-release ${dry_run_opt} | tee semantic-output.txt

I'm not sure it's gonna work, I added a unit test specific to gitlab to ensure that the current authentication provided by CI_REPOSITORY_URL is replaced by gitlab-ci-token:${GITLAB_TOKEN} and it already works so. But we never know !

syphernl commented 13 hours ago

I have tested without the --repository-url but it makes no difference, it still fails with the same error.
It does appear I can omit that argument from my template though, because Semantic-Release gets it from package.json and/or generates it since the behavior is still the same.

kilianpaquier commented 11 hours ago

Yes, by default repository-url is either retrieved from package.json or from git origin URL (which in CI is expected to exist).

If not too confidential too, could you provide your .releaserc configuration so I can try reproducing the error on my end with the same conditions ?

syphernl commented 11 hours ago

Here is our .releaserc.js:

module.exports = {
  plugins: [
    [
      '@semantic-release/commit-analyzer',
      {
        preset: 'angular',
        parserOpts: {
          noteKeywords: ['BREAKING CHANGE', 'BREAKING CHANGES', 'BREAKING']
        },
        releaseRules: [
          { type: 'build', release: 'patch' },
          { type: 'deploy', release: 'patch' },
          { type: 'chore', release: 'patch' },
          { type: 'docs', release: 'patch' },
          { type: 'test', release: 'patch' },
          { type: 'style', release: 'patch' },
          { type: 'ci', release: 'patch' }
        ]
      }
    ],
    [
      '@semantic-release/release-notes-generator',
      {
        preset: 'conventionalcommits',
        parserOpts: {
          noteKeywords: ['BREAKING CHANGE', 'BREAKING CHANGES', 'BREAKING']
        },
        writerOpts: {
          commitsSort: ['subject', 'scope']
        },
        presetConfig: {
          types: [
            { type: 'feat', section: 'Features' },
            { type: 'fix', section: 'Bug Fixes' },
            { type: 'chore', section: 'Chores', hidden: false },
            { type: 'refactor', section: 'Internal', hidden: false },
            { type: 'perf', section: 'Performance', hidden: false },
            { type: 'docs', section: 'Documentation', hidden: false },
            { type: 'ci', section: 'DevOps', hidden: false },
            { type: 'test', section: 'Tests', hidden: false }
          ]
        }
      }
    ],
    [
      '@bpgeck/semantic-release-kaniko',{ }
    ],
    [
      '@semantic-release/gitlab',
      {
        successComment: ':tada: This ${issue ? "issue" : "MR"} is included in version [v${nextRelease.version}](${releases.find(release => /gitlab\\.com/i.test(release.url)).url}).',
        labels: 'semantic-release'
      }
    ],
    [
      'semantic-release-slack-bot',
      {
        notifyOnSuccess: true,
        markdownReleaseNotes: true,
        onSuccessTemplate: {
          blocks: [
            {
              type: 'section',
              text: {
                type: 'mrkdwn',
                text: `A new version of \`$package_name\` has been released!\nCurrent version is *$npm_package_version*`
              }
            },
            { type: 'divider' },
            {
              type: 'section',
              text: {
                type: 'mrkdwn',
                text: '$release_notes'
              }
            }
          ],
          attachments: [
            {
              blocks: [
                {
                  type: 'context',
                  elements: [
                    {
                      type: 'mrkdwn',
                      text: `:link: <${process.env.CI_ENVIRONMENT_URL}|View ${process.env.CI_ENVIRONMENT_NAME} website>`
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    ],
    [
      "@kilianpaquier/semantic-release-backmerge",
      {
        targets: [
          { from: "production", to: "develop" },
          { from: "acceptance", to: "develop" }
        ],
        title: "ci(release): fix mergeback from ${ from } into ${ to }",
      }
    ],
  ],
  branches: [
    { name: 'main', prerelease: true },
    { name: 'review', prerelease: true },
    { name: 'acceptance', prerelease: true },
    { name: 'production', prerelease: false }
  ]
};

I have also reported the issue with marked-terminal to the Semantic-Release project since that particular problem probably lies there.

This is such a strange issue..

Project repo: fails
POC repo (I created before applying the setup to our project repo: works fine
Fork of the project repo: fails
Newly created "barebones" project: works

It feels like that there is some (UI-)hidden GitLab setting that is active on the project repo and was also added to the fork.. My next step to try is to create a fresh project (no fork) and adding in the project code to see what happens..

syphernl commented 8 hours ago

My next step to try is to create a fresh project (no fork) and adding in the project code to see what happens..

Sadly, this causes the same problem..

Created new a fresh project on GitLab, cloned locally (as develop)
Exported the original repo (as tgz via the GitLab UI), extracted in the new repo folder. Add files, commit (with [skip ci])
Created Access Token with appropriate access scopes
Created acceptance and production branches based on develop
Made them protected (only Maintainer + Access Token can access it)
Made a "hotfix" on the acceptance branch
Backmerge failed with same ENOENT error

My investigation continues..

kilianpaquier commented 7 hours ago

Hi !

Could we clarify a little the default values for protected branches accesses for your projects ?

Here's an example for semantic-release-testing:

syphernl commented 6 hours ago

The "1 role" is Maintainer and the "1 user" is the Access Token (which is also a Maintainer, so having it explicitly added is a bit redundant)

I am currently testing with some patches to your code to drop in some crude debug logging to figure out what inputs it uses in its functions etc.

syphernl commented 6 hours ago

I think I'm on to something. I added a (very crude) check in verify-config to run git --version and check whether that command works..

Result:

[12:43:09 PM] [semantic-release] › ✘  An error occurred while running semantic-release: Error: Git binary is not installed or not found in PATH.

That likely explains the ENOENT... However, it does not explain why this is happening.

The configured PATH (default) is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

git exists in /usr/bin/git which is included in the path so the git command should just work..

kilianpaquier commented 6 hours ago

It's weird I agree since in the .gitlab-ci.yml job, you're executing git commands just before ...

But it would explain things, it seems ENOENT indicates missing binary in Alpine OS. How is installed git with your custom Docker image ?

syphernl commented 6 hours ago

Perhaps my check is not working as it should though (I'm not a JS dev 😅 ) since running it with the explicit path still reports that it is missing..

But it would explain things, it seems ENOENT indicates missing binary in Alpine OS. How is installed git with your custom Docker image ?

Not doing anything special for that. Our baseimage is node:20.18.0-alpine3.20 and Git gets installed via apk: RUN apk add --update --no-cache git

kilianpaquier commented 6 hours ago

Hidden

kilianpaquier commented 6 hours ago

Do you want me to check it ? You can open a draft pull request 🙂.

Checking on my end with an alpine image instead of node:lts-slim. Well doesn't work on my end but with another error, the CI doesn't handle Alpine images ahah https://gitlab.com/kilianpaquier/semantic-release-testing/-/jobs/8153546219.

Could you try on your end to use node:lts-slim ?

syphernl commented 5 hours ago

Could you try on your end to use node:lts-slim ?

That is something I will test, however it will require quite some extensive rebuilding of the Dockerfile since we install a bunch of dependencies which are included in Alpine and not with Debian. For actual usage we do prefer to keep using Alpine though. Not just because they are smaller in size, also because the footprint is smaller and results in less potential security issues (have to keep the security scanning jobs happy 😉 )

FWIW: the original Alpine-based image is 472MB while the new one is 790MB so it is quite a difference size-wise ;-)

syphernl commented 5 hours ago

The Debian-based image is running in to the very same ENOENT issue.. (git version 2.39.5).

I did see in my added debug logging that the git fetch is being done using the CI_JOB_TOKEN. Could this somehow be a problem? Perhaps the GITLAB_TOKEN could/should be used instead?

kilianpaquier commented 5 hours ago

Yes, all git commands should be made with the GITLAB_TOKEN (repository URL is parsed and authentication is replaced, maybe too late ?). It could explain the error.

syphernl commented 4 hours ago

Yes, all git commands should be made with the GITLAB_TOKEN (repository URL is parsed and authentication is replaced, maybe too late ?). It could explain the error.

The repository URL that gets passed along to fetch function contains a glcbt-xxx password which is a (temporary) job token and not the glpat-xxx token I have configured as GITLAB_TOKEN.

kilianpaquier commented 4 hours ago

Let me fix that (found the invalid code) hoping it will be enough to fix the root issue 🙂

kilianpaquier / semantic-release-backmerge

Failed to backmerge branches.: xJ: Command failed with ENOENT: git fetch #17