var-transform with '_' in replace

mesaglio commented 9 months ago

Hi! @Elizafox I dont know if this functionality is not working as spected, or is not for this case.

I tested locally with the following test. This generate this error

2024/01/19 11:43:58 INFO   evaluated: http://127.0.0.1:52223/wine/cheese/cheese-7.0.1+1.tar.gz
    bump_test.go:118:
            Error Trace:    /Users/jmesaglio/Documents/repos/melange/pkg/renovate/bump/bump_test.go:118
                                        /usr/local/go/src/net/http/server.go:2136
                                        /usr/local/go/src/net/http/server.go:2938
                                        /usr/local/go/src/net/http/server.go:2009
                                        /usr/local/go/src/runtime/asm_arm64.s:1197
            Error:          Not equal:
                            expected: "/wine/cheese/cheese-7.0.1+1.tar.gz"
                            actual  : "/wine/cheese/cheese-7.0.1.tar.gz"

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1 +1 @@
                            -/wine/cheese/cheese-7.0.1+1.tar.gz
                            +/wine/cheese/cheese-7.0.1.tar.gz
            Test:           TestBump_versions/major_minor_patch.yaml#01

I undestand the var transformation, generate same-package-version with 7.0.1 value.

var-transforms:
  - from: ${{package.version}}
    match: (\d.\d.\d)
    replace: $1
    to: same-package-version

We found this error en melanga package when uses fetch node.

For example:

package:
  name: temurin-11-jdk
  version: "11.0.21.9"
  epoch: 0
  description: "Adoptium Temurin 11 JDK"
  copyright:
    - license: Apache-2.0
  dependencies:
    runtime:

environment:
  contents:
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    repositories:
      - https://packages.wolfi.dev/os
    packages:
      - wolfi-base

var-transforms:
  - from: ${{package.version}}
    match: \.(\d+)$
    replace: _$1
    to: mangled-version-binary
  - from: ${{package.version}}
    match: \.(\d+)$
    replace: +$1
    to: mangled-name-jdk

pipeline:
  - runs: |
      mkdir -p "${{targets.destdir}}/jdk"

  - if: ${{build.arch}} == 'x86_64'
    uses: fetch
    with:
      extract: false
      uri: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-${{vars.mangled-name-jdk}}/OpenJDK11U-jdk_x64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz
      expected-sha256: 60ea98daa09834fdd3162ca91ddc8d92a155ab3121204f6f643176ee0c2d0d5e

  - if: ${{build.arch}} == 'x86_64'
    runs: |
      tar -xf OpenJDK11U-jdk_x64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz --strip-components=1 -C "${{targets.destdir}}/jdk"

  - if: ${{build.arch}} == 'aarch64'
    uses: fetch
    with:
      extract: false
      uri: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-${{vars.mangled-name-jdk}}/OpenJDK11U-jdk_aarch64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz
      expected-sha256: 8c3146035b99c55ab26a2982f4b9abd2bf600582361cf9c732539f713d271faf

  - if: ${{build.arch}} == 'aarch64'
    runs: |
      tar -xf OpenJDK11U-jdk_aarch64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz --strip-components=1 -C "${{targets.destdir}}/jdk"

  - uses: strip

update:
  enabled: true
  ignore-regex-patterns:
    - ".*-beta"
  shared: true
  github:
    identifier: adoptium/temurin11-binaries
    strip-prefix: jdk-
    tag-filter: jdk-11.
    use-tag: false

When I run the wolfictl update --release-monitoring-query=false, we look the evaluated uri dont have the var transformations results.

2024/01/21 21:28:59 attempting to bump version to 11.0.22+7
2024/01/21 21:28:59 processing fetch node:
2024/01/21 21:28:59   uri: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-${{vars.mangled-name-jdk}}/OpenJDK11U-jdk_x64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz
2024/01/21 21:28:59   evaluated: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22+7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.22+7.tar.gz
2024/01/21 21:29:00 wolfictl update: temurin-11-jdk: existing issue 439 already exists for error message: failed to bump package temurin-11-jdk to version 11.0.22+7: got 404 Not Found when fetching https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22+7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.22+7.tar.gz

If we run a melange build the vars transformation generate the correct data

mesaglio commented 9 months ago

@jonjohnsonjr @vaikas can help with this?

jonjohnsonjr commented 9 months ago

When I run the wolfictl update --release-monitoring-query=false, we look the evaluated uri dont have the var transformations results.

Sorry I'm a little slow, what do you expect this line to be?

2024/01/21 21:28:59   evaluated: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22+7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.22+7.tar.gz

vaikas commented 9 months ago

zomg, I'm so sorry I missed this earlier. So looks like it may be that the URL unfortunately needs maybe more munging, because the release here: https://github.com/adoptium/temurin11-binaries/releases/tag/jdk-11.0.22%2B7

Points to this URL: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22%2B7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.22_7.tar.gz

Looks like the + is there in the path, but not in the filename?

If I change the URL that is being constructed to this (change the last + to _): https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22+7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.22_7.tar.gz

It works. So looks like there are two mungings that need to happen, one that leaves the + in it, so you'll have: jdk-11.0.22+7 and the filename should be: linux_hotspot_11.0.22_7.tar.gz

mesaglio commented 9 months ago

Yes. This mungings is not setting with this var-transformation?

var-transforms:
  - from: ${{package.version}}
    match: \.(\d+)$
    replace: _$1
    to: mangled-version-binary
  - from: ${{package.version}}
    match: \.(\d+)$
    replace: +$1
    to: mangled-name-jdk

Replacing the last . with _ and +, respectively? If I make a melange build this url is generating correctly, but no when I run a wolfi update.

vaikas commented 9 months ago

I may have misread it, but it seemed to me like one of the version is getting correctly munged, but not the second one?

mesaglio commented 9 months ago

This isn't a melange issue, I created an incorrect regex to golang implementation.

Now I understand the regex in that package need to match all, from left to right. Here is un example. So I add + to the melange version string and update the regexes.

My bad :yum: , thanks for your help :raised_hands:!

vaikas commented 9 months ago

You can't use the + in the melange version string (if I understand correctly what you are proposing on doing) https://github.com/chainguard-dev/melange/blob/main/docs/VAR-TRANSFORMS.md

mesaglio commented 9 months ago

Good point thanks. Yes I saw that documentation, but I think in this case temurin have a more tricky version standar. They can generate 11.0.21+9 and 11.0.21+9.1, so the regex is a little different. Finally, I think have the final regex version.

I may have misread it, but it seemed to me like one of the version is getting correctly munged, but not the second one?

This happend because the replace function have a distinct functionality if I use $1_$2 and $1+$2. Poc

If you need to concat with _, changing $1 to ${1} works (https://github.com/golang/go/issues/32885).

So the final melange vars transformation, are

var-transforms:
  # This is needed because are version with +7.1 but in github is +7
  - from: ${{package.version}}
    match: ^(\d+\.\d+\.\d+).(\d+).*
    replace: "$1+$2"
    to: mangled-package-version
  # This is needed because are version with +7.1 but in github is _7
  - from: ${{package.version}}
    match: ^(\d+\.\d+\.\d+).(\d+).*
    replace: "${1}_${2}"
    to: mangled-version-binary

Thanks for all @vaikas 💪 !

vaikas commented 9 months ago

I am still a bit curious if the interaction between wolfictl & melange will cause grief, since I'm thinking that wolfictl will try to bump to invalid melange version string of say: 11.0.22+7.1. For example, I grabbed the file above, and running wolfictl check update temurin-11-jdk.yaml spits out this:

ℹ️            | error during command execution: package temurin-11-jdk: update found newer version 11.0.22+7.1 compared with package.version 11.0.21.9 in melange config

And if all it does is change the package.version to that it is not going to work. But I may have misunderstood something 😆

mesaglio commented 9 months ago

Another good point!! I think in this type case, we need to add version-transform in the Update section.

Without this the next update pr will be like this

But if we add this version-transform

version-transform:
    - match: ^(.*)\+(.*)$
      replace: "${1}.${2}"

2024/01/31 22:55:35 wolfictl update: there is a new stable version available temurin-11-jdk, current wolfi version 11.0.21.9, new 11.0.22.7.1

It would be

👍

The final file

package:
  name: temurin-11-jdk
  version: "11.0.21.9"
  epoch: 0
  description: "Adoptium Temurin 11 JDK"
  copyright:
    - license: Apache-2.0
  dependencies:
    runtime:

environment:
  contents:
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    repositories:
      - https://packages.wolfi.dev/os
    packages:
      - wolfi-base

var-transforms:
  # This is needed because are version with +7.1 but in github is +7
  - from: ${{package.version}}
    match: ^(\d+\.\d+\.\d+).(\d+).*
    replace: "$1+$2"
    to: mangled-package-version
  # This is needed because are version with +7.1 but in github is _7
  - from: ${{package.version}}
    match: ^(\d+\.\d+\.\d+).(\d+).*
    replace: "${1}_${2}"
    to: mangled-version-binary

pipeline:
  - runs: |
      mkdir -p "${{targets.destdir}}/jdk"

  - if: ${{build.arch}} == 'x86_64'
    uses: fetch
    with:
      extract: false
      uri: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-${{vars.mangled-package-version}}/OpenJDK11U-jdk_x64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz
      expected-sha256: 60ea98daa09834fdd3162ca91ddc8d92a155ab3121204f6f643176ee0c2d0d5e

  - if: ${{build.arch}} == 'x86_64'
    runs: |
      tar -xf OpenJDK11U-jdk_x64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz --strip-components=1 -C "${{targets.destdir}}/jdk"

  - if: ${{build.arch}} == 'aarch64'
    uses: fetch
    with:
      extract: false
      uri: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-${{vars.mangled-package-version}}/OpenJDK11U-jdk_aarch64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz
      expected-sha256: 8c3146035b99c55ab26a2982f4b9abd2bf600582361cf9c732539f713d271faf

  - if: ${{build.arch}} == 'aarch64'
    runs: |
      tar -xf OpenJDK11U-jdk_aarch64_linux_hotspot_${{vars.mangled-version-binary}}.tar.gz --strip-components=1 -C "${{targets.destdir}}/jdk"

  - uses: strip

update:
  enabled: true
  ignore-regex-patterns:
    - ".*-beta"
  shared: true
  github:
    identifier: adoptium/temurin11-binaries
    strip-prefix: jdk-
    tag-filter: jdk-11.
    use-tag: false
  version-transform:
    - match: ^(.*)\+(.*)$
      replace: "${1}.${2}"

vaikas commented 9 months ago

Woohoo!!! Well, that was easy 🤣 Nice work!

vaikas commented 9 months ago

Since all this is fresh in your mind, and you are an expert now!!! Would you mind crafting an update to the following documentation files with these examples?

https://github.com/chainguard-dev/melange/blob/main/docs/VAR-TRANSFORMS.md

Adding a "reverse" of the update example that you have to do support the java? https://github.com/chainguard-dev/melange/blob/main/docs/UPDATE.md

Having a complete example of going from package.version => fetchable URL, as well as how to add support for the reverse (supporting automatic updates with wolfictl) would be super nice to have!!

chainguard-dev / melange

var-transform with '_' in replace #945