search

renovatebot / config-help

Please use the Discussions feature of https://github.com/renovatebot/renovate instead
https://github.com/renovatebot/renovate/discussions
MIT License
27 stars 16 forks source link

Versioning regex contains HTML entities breaking handlebars #736

Closed nicholasdille closed 4 years ago

nicholasdille commented 4 years ago

Which Renovate are you using?

Renovate Open Source CLI (version 21.21.8)

Which platform are you using?

GitLab self-hosted

Have you checked the logs? Don't forget to include them if relevant

Yes, full log is included at the bottom.

What would you like to do?

We are using a regex manager to parse the following Dockerfile:

FROM docker:19.03.12 AS base
# renovate: datasource=github-releases depName=kubernetes-sigs/kustomize versioning=regex:^kustomize/v(?<major>\d+)\.(?<minor>\d+)\.(?<patch>\d+)$
ENV KUSTOMIZE_VERSION="kustomize/v3.6.1"
RUN curl --verbose --location --fail https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/${KUSTOMIZE_VERSION#kustomize/}/kustomize_${KUSTOMIZE_VERSION#kustomize/}_linux_amd64.tar.gz | \
        tar -xvzC /usr/local/bin/ kustomize

Apparently, the regex contained in the versioning field of the comment does not work because the < and > characters are replaced with HTML entities &lt; and %gt;. Following is the relevant part of the log:

 INFO: Repository has invalid config (repository=***********/Test/renovate-versioning-regex)
       "error": {
         "configFile": "^kustomize/v(?&lt;major&gt;\\d+)\\.(?&lt;minor&gt;\\d+)\\.(?&lt;patch&gt;\\d+)$",
         "validationError": "regex versioning needs at least one major, minor or patch group defined"
       }

The configuration file contains the following regex manager:

{
# REDACTED
         "regexManagers": [
           {
             "fileMatch": ["^Dockerfile$"],
             "matchStrings": [
               "#\\s*renovate:\\s*datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\sENV .*?_VERSION=(?<currentValue>.*)\\s"
             ],
             "versioningTemplate": "{{#if versioning}}{{versioning}}{{else}}semver{{/if}}"
           }
         ]
}

How can we make this work? Thanks in advance!

Full log

DEBUG: File config
       "config": {}
DEBUG: CLI config
       "config": {
         "repositories": ["***********/Test/renovate-versioning-regex"],
         "logLevel": "debug",
         "platform": "gitlab",
         "endpoint": "https://gitlab.***********/api/v4",
         "token": "***********",
         "autodiscover": false
       }
DEBUG: Env config
       "config": {"hostRules": []}
DEBUG: Combined config
       "config": {
         "hostRules": [],
         "repositories": ["***********/Test/renovate-versioning-regex"],
         "logLevel": "debug",
         "platform": "gitlab",
         "endpoint": "https://gitlab.***********/api/v4",
         "token": "***********",
         "autodiscover": false
       }
DEBUG: Adding trailing slash to endpoint
DEBUG: Using platform gitAuthor: *********** <***********@***********.com>
DEBUG: Using baseDir: /tmp/renovate
DEBUG: Using cacheDir: /tmp/renovate/cache
DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
DEBUG: Limits.init enter method
DEBUG: Limits.init prCommitsPerRunLimit processing
DEBUG: Limits.init prCommitsPerRunLimit variable is not set. Ignoring prCommitsPerRunLimit
 INFO: Repository started (repository=***********/Test/renovate-versioning-regex)
       "renovateVersion": "21.21.8"
DEBUG: Using localDir: /tmp/renovate/repos/gitlab/***********/Test/renovate-versioning-regex (repository=***********/Test/renovate-versioning-regex)
DEBUG: ***********/Test/renovate-versioning-regex default branch = master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Bot email=***********@***********.com (repository=***********/Test/renovate-versioning-regex)
DEBUG: Enabling Git FS (repository=***********/Test/renovate-versioning-regex)
DEBUG: ***********/Test/renovate-versioning-regex http URL = https://gitlab.***********/***********/test/renovate-versioning-regex.git (repository=***********/Test/renovate-versioning-regex)
DEBUG: Initializing git repository into /tmp/renovate/repos/gitlab/***********/Test/renovate-versioning-regex (repository=***********/Test/renovate-versioning-regex)
DEBUG: git clone completed (repository=***********/Test/renovate-versioning-regex)
       "durationMs": 408
DEBUG: latest commit (repository=***********/Test/renovate-versioning-regex)
       "latestCommitDate": "2020-07-03 11:21:12 +0200"
DEBUG: Setting git author name (repository=***********/Test/renovate-versioning-regex)
       "gitAuthorName": "***********"
DEBUG: Setting git author email (repository=***********/Test/renovate-versioning-regex)
       "gitAuthorEmail": "***********@***********.com"
DEBUG: resetMemCache() (repository=***********/Test/renovate-versioning-regex)
DEBUG: detectSemanticCommits() (repository=***********/Test/renovate-versioning-regex)
DEBUG: getCommitMessages (repository=***********/Test/renovate-versioning-regex)
DEBUG: Semantic commits detection: unknown (repository=***********/Test/renovate-versioning-regex)
DEBUG: No semantic commits detected (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: latest commit (repository=***********/Test/renovate-versioning-regex)
       "branchName": "master",
       "latestCommitDate": "2020-07-03 11:21:12 +0200"
DEBUG: checkOnboarding() (repository=***********/Test/renovate-versioning-regex)
DEBUG: isOnboarded() (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(renovate.json) (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(renovate.json5) (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(.github/renovate.json) (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(.github/renovate.json5) (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(.gitlab/renovate.json) (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(.gitlab/renovate.json5) (repository=***********/Test/renovate-versioning-regex)
DEBUG: findFile(.renovaterc) (repository=***********/Test/renovate-versioning-regex)
DEBUG: config file exists (repository=***********/Test/renovate-versioning-regex)
DEBUG: ensureIssueClosing() (repository=***********/Test/renovate-versioning-regex)
DEBUG: Repo is onboarded (repository=***********/Test/renovate-versioning-regex)
DEBUG: Found .renovaterc config file (repository=***********/Test/renovate-versioning-regex)
DEBUG: Repository config (repository=***********/Test/renovate-versioning-regex)
       "configFile": ".renovaterc",
       "config": {
         "extends": [
           "config:base",
           "docker:enableMajor",
           ":disableRateLimiting",
           ":automergePr",
           ":automergeRequireAllStatusChecks",
           ":semanticCommits",
           ":maintainLockFilesWeekly"
         ],
         "baseBranches": ["master"],
         "regexManagers": [
           {
             "fileMatch": ["^Dockerfile$"],
             "matchStrings": [
               "#\\s*renovate:\\s*datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\sENV .*?_VERSION=(?<currentValue>.*)\\s"
             ],
             "versioningTemplate": "{{#if versioning}}{{versioning}}{{else}}semver{{/if}}"
           }
         ]
       }
DEBUG: migrateAndValidate() (repository=***********/Test/renovate-versioning-regex)
DEBUG: No config migration necessary (repository=***********/Test/renovate-versioning-regex)
DEBUG: massaged config (repository=***********/Test/renovate-versioning-regex)
       "config": {
         "extends": [
           "config:base",
           "docker:enableMajor",
           ":disableRateLimiting",
           ":automergePr",
           ":automergeRequireAllStatusChecks",
           ":semanticCommits",
           ":maintainLockFilesWeekly"
         ],
         "baseBranches": ["master"],
         "regexManagers": [
           {
             "fileMatch": ["^Dockerfile$"],
             "matchStrings": [
               "#\\s*renovate:\\s*datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\sENV .*?_VERSION=(?<currentValue>.*)\\s"
             ],
             "versioningTemplate": "{{#if versioning}}{{versioning}}{{else}}semver{{/if}}"
           }
         ]
       }
DEBUG: migrated config (repository=***********/Test/renovate-versioning-regex)
       "config": {
         "extends": [
           "config:base",
           "docker:enableMajor",
           ":disableRateLimiting",
           ":automergePr",
           ":automergeRequireAllStatusChecks",
           ":semanticCommits",
           ":maintainLockFilesWeekly"
         ],
         "baseBranches": ["master"],
         "regexManagers": [
           {
             "fileMatch": ["^Dockerfile$"],
             "matchStrings": [
               "#\\s*renovate:\\s*datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\sENV .*?_VERSION=(?<currentValue>.*)\\s"
             ],
             "versioningTemplate": "{{#if versioning}}{{versioning}}{{else}}semver{{/if}}"
           }
         ]
       }
DEBUG: Found repo ignorePaths (repository=***********/Test/renovate-versioning-regex)
       "ignorePaths": [
         "**/node_modules/**",
         "**/bower_components/**",
         "**/vendor/**",
         "**/examples/**",
         "**/__tests__/**",
         "**/test/**",
         "**/tests/**",
         "**/__fixtures__/**"
       ]
DEBUG: checkBaseBranch() (repository=***********/Test/renovate-versioning-regex)
DEBUG: config.repoIsOnboarded=true (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: latest commit (repository=***********/Test/renovate-versioning-regex)
       "branchName": "master",
       "latestCommitDate": "2020-07-03 11:21:12 +0200"
DEBUG: Setting branchPrefix: renovate/ (repository=***********/Test/renovate-versioning-regex)
DEBUG: No vulnerability alerts found (repository=***********/Test/renovate-versioning-regex)
DEBUG: processRepo() (repository=***********/Test/renovate-versioning-regex)
DEBUG: baseBranches (repository=***********/Test/renovate-versioning-regex)
       "baseBranches": ["master"]
DEBUG: baseBranch: master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: latest commit (repository=***********/Test/renovate-versioning-regex)
       "branchName": "master",
       "latestCommitDate": "2020-07-03 11:21:12 +0200"
DEBUG: extract() (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)tasks/[^/]+\.ya?ml$ for manager ansible (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)requirements\.ya?ml$ for manager ansible-galaxy (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)WORKSPACE(|\.bazel)$ for manager bazel (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.bzl$ for manager bazel (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: buildkite\.ya?ml for manager buildkite (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.buildkite/.+\.ya?ml$ for manager buildkite (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)Gemfile$ for manager bundler (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)Cargo.toml$ for manager cargo (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/).circleci/config.yml$ for manager circleci (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)Podfile$ for manager cocoapods (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)([\w-]*)composer.json$ for manager composer (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)deps\.edn$ for manager deps-edn (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)docker-compose[^/]*\.ya?ml$ for manager docker-compose (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/|\.)Dockerfile$ for manager dockerfile (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)Dockerfile\.[^/]*$ for manager dockerfile (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/).drone.yml$ for manager droneci (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/).gitmodules$ for manager git-submodules (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^\.github/main.workflow$ for manager github-actions (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^\.github/workflows/[^/]+\.ya?ml$ for manager github-actions (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^\.gitlab-ci\.yml$ for manager gitlabci (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^\.gitlab-ci\.yml$ for manager gitlabci-include (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)go.mod$ for manager gomod (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.gradle(\.kts)?$ for manager gradle (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)gradle.properties$ for manager gradle (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)gradle/wrapper/gradle-wrapper.properties$ for manager gradle-wrapper (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)requirements.yaml$ for manager helm-requirements (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)values.yaml$ for manager helm-values (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)helmfile.yaml$ for manager helmfile (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^Formula/[^/]+[.]rb$ for manager homebrew (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.html?$ for manager html (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)kustomization\.yaml for manager kustomize (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)project\.clj$ for manager leiningen (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.pom\.xml$ for manager maven (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)pom\.xml$ for manager maven (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)package.js$ for manager meteor (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)mix\.exs$ for manager mix (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^.node-version$ for manager nodenv (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)package.json$ for manager npm (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.(?:cs|fs|vb)proj$ for manager nuget (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.(?:props|targets)$ for manager nuget (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.config\/dotnet-tools\.json$ for manager nuget (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)\.nvmrc$ for manager nvm (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)([\w-]*)requirements.(txt|pip)$ for manager pip_requirements (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)setup.py$ for manager pip_setup (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)Pipfile$ for manager pipenv (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)pyproject\.toml$ for manager poetry (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)pubspec\.ya?ml$ for manager pub (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^Dockerfile$ for manager regex (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)\.ruby-version$ for manager ruby-version (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.sbt$ for manager sbt (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: project/[^/]*.scala$ for manager sbt (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: (^|/)Package\.swift for manager swift (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: \.tf$ for manager terraform (repository=***********/Test/renovate-versioning-regex)
DEBUG: Using file match: ^.travis.yml$ for manager travis (repository=***********/Test/renovate-versioning-regex)
DEBUG: Matched 1 file(s) for manager dockerfile: Dockerfile (repository=***********/Test/renovate-versioning-regex)
DEBUG: Matched 1 file(s) for manager regex: Dockerfile (repository=***********/Test/renovate-versioning-regex)
DEBUG: Found a multistage build stage name (repository=***********/Test/renovate-versioning-regex)
DEBUG: Found dockerfile package files (repository=***********/Test/renovate-versioning-regex)
DEBUG: Found regex package files (repository=***********/Test/renovate-versioning-regex)
DEBUG: Found 2 package file(s) (repository=***********/Test/renovate-versioning-regex)
 INFO: Dependency extraction complete (repository=***********/Test/renovate-versioning-regex)
       "baseBranch": "master",
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 1, "depCount": 1},
           "regex": {"fileCount": 1, "depCount": 1}
         },
         "total": {"fileCount": 2, "depCount": 2}
       }
DEBUG: baseBranch: master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: Setting baseBranch to master (repository=***********/Test/renovate-versioning-regex)
DEBUG: latest commit (repository=***********/Test/renovate-versioning-regex)
       "branchName": "master",
       "latestCommitDate": "2020-07-03 11:21:12 +0200"
 INFO: Repository has invalid config (repository=***********/Test/renovate-versioning-regex)
       "error": {
         "configFile": "^kustomize/v(?&lt;major&gt;\\d+)\\.(?&lt;minor&gt;\\d+)\\.(?&lt;patch&gt;\\d+)$",
         "validationError": "regex versioning needs at least one major, minor or patch group defined"
       }
DEBUG: raiseConfigWarningIssue() (repository=***********/Test/renovate-versioning-regex)
DEBUG: getBranchPr(renovate/configure) (repository=***********/Test/renovate-versioning-regex)
DEBUG: ensureIssue() (repository=***********/Test/renovate-versioning-regex)
DEBUG: Updating issue body (repository=***********/Test/renovate-versioning-regex)
DEBUG: Repository timing splits (milliseconds) (repository=***********/Test/renovate-versioning-regex)
       "splits": {"init": 1952, "extract": 1305},
       "total": 3875
DEBUG: http statistics (repository=***********/Test/renovate-versioning-regex)
       "hostStats": ["gitlab.***********, 5 requests, 162ms average"],
       "totalRequests": 5
 INFO: Repository finished (repository=***********/Test/renovate-versioning-regex)
       "durationMs": 3875
DEBUG: Renovate exiting
DEBUG: getLabels(https://index.docker.io, library/docker, latest)
DEBUG: getManifestResponse(https://index.docker.io, library/docker, latest)
rarkins commented 4 years ago

Hi, please create a reproduction of this in a public github.com repo so that we can debug it.

nicholasdille commented 4 years ago

@rarkins, thanks you for the fast reply. I have created this repo to reproduce the issue. RenovateBot has already created issue #1 complaining about this.

Let me know if you need anything else!

rarkins commented 4 years ago

The escaping comes from handlebars. If you use {{{versioning}}} with three brackets then the problem is resolved. However the currentValue match is not quite correct so it still fails without removing the brackets around version:

    {
      "fileMatch": ["^Dockerfile$"],
      "matchStrings": [
        "#\\s*renovate:\\s*datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\sENV .*?_VERSION=\"(?<currentValue>.*)\"\\s"
      ],
      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
    }

It's not picking up the v3.7.0 update though - I'll leave that to you to troubleshoot.

rarkins commented 4 years ago

Seems that the API returns something different to what you see in the UI: https://api.github.com/repos/kubernetes-sigs/kustomize/releases

rarkins commented 4 years ago

The UI mixes releases and tags. There's a kustomize/v3.7.0 tag but not a release. Latest is v3.6.1: image

nicholasdille commented 4 years ago

Thanks for the help! Did I miss this in the documentation?

I am still struggeling with finding a supported value in the Dockerfile. Do I have to enter the version (3.6.1) or the tag_name from the release (kustomize/3.6.1)?

rarkins commented 4 years ago

Thanks for the help! Did I miss this in the documentation?

The docs mention that handlebars is used to compile templates and it was handlebars that was escaping the special characters. I have just now updated the example you copied from to use triple braces now and added an explicit note in a couple of places to recommend triple braces.

I am still struggeling with finding a supported value in the Dockerfile. Do I have to enter the version (3.6.1) or the tag_name from the release (kustomize/3.6.1)?

It needs to match with the convention that the repo uses, which is kustomize/v3.6.1. i.e. what you had last time. The problem is that kustomize/v3.7.0 is not a release. I don't know if that's a mistake of the maintainers or if it was intention to make them tags only.

If you change from github-releases to github-tags then you will get a PR: https://github.com/renovate-tests/renovate-versioning-regex/pull/2/files

nicholasdille commented 4 years ago

Yes, switching to github-tags allowed the update to kustomize/v3.7.0. In addition, I had to remove the double quotes around the value in KUSTOMIZE_VERSION... in the end not suprising... that's what the regex says ;-)

This state works! Thanks!

rarkins commented 4 years ago

I'd worked past the "" in my above config actually, but glad you worked it out independently.

nicholasdille commented 4 years ago

Your help was very much appreciated