Track problems in Envoy 1.22.0

[codefromthecrypt]

I clicked Run Workflow for v1.22.0 and v1.22.0_debug on https://github.com/tetratelabs/archive-envoy/actions/workflows/release.yaml

Only windows worked.

• The linux image changed, so scripts failed to find the binaries
• Envoy 1.22.0 hasn't made it to OS/x due to drift https://github.com/Homebrew/homebrew-core/pull/99389

Next steps:

• figure out drift in linux image and make a change to the shell script in a way that doesn't break versions before 1.22. re-run v1.22.0 and v1.22.0_debug workflows after that workaround is merged.
• re-run workflow v1.22.0 once the 1.22.0 formula is fixed in homebrew

[Lord-Y]

Waiting for the linux version :)

[codefromthecrypt]

Update:

• no update on homebrew as no-one has attempted to add 1.22
• below are some breadcrumbs for linux image drift

Looks like the image layer changed in 1.22.0 (for linux). It is still the same in windows and <1.22 linux

$ sh -x ./bin/archive_release_version.sh envoyproxy/envoy v1.22.0 check
--snip--
using dist: envoy-v1.22.0-linux-amd64
+ set +e
+ ./bin/car_envoy.sh v1.22.0 linux amd64 list v1.22.0/envoy-v1.22.0-linux-amd64
error: usr/local/bin/envoy not found in layer
--snip--
$ sh -x ./bin/car_envoy.sh v1.22.0 linux amd64 list v1.22.0/envoy-v1.22.0-linux-amd64
--snip--
+ reference=envoyproxy/envoy:v1.22.0
+ car -t --platform linux/amd64 --created-by-pattern 'ADD linux' --strip-components 2 -qf envoyproxy/envoy:v1.22.0 usr/local/bin/envoy
error: usr/local/bin/envoy not found in layer
--snip--
$ car -t --platform linux/amd64 --strip-components 2 -vvf envoyproxy/envoy:v1.22.0 usr/local/bin/envoy
--snip--
https://index.docker.io/v2/envoyproxy/envoy/blobs/sha256:4086e842c0d1e552c62d8b94d92cb0c5a91fbccf62284f1e1c60d60a182270fa size=21351994
CreatedBy: COPY /usr/local/bin/envoy* /usr/local/bin/ # buildkit
-rwxr-xr-x  59397736    Apr 16 01:42:42 usr/local/bin/envoy
--snip--
$ car -t --platform linux/amd64 --created-by-pattern '/usr/local/bin/envoy' --strip-components 2 -qf envoyproxy/envoy:v1.22.0 
usr/local/bin/envoy

So what happened was the image layers changed in 1.22. To fix this requires a patch. I've put this here as for the sake of project continuity someone besides me needs to become familiar with the project, even if a volunteer. PR welcome!

diff --git a/bin/car_envoy.sh b/bin/car_envoy.sh
index ea1a34f..05d9353 100755
--- a/bin/car_envoy.sh
+++ b/bin/car_envoy.sh
@@ -110,7 +110,15 @@ linux)
     reference=envoyproxy/envoy:${version}
   fi

-  ${car} --created-by-pattern 'ADD linux' --strip-components 2 -qf "${reference}" ${files}
+  # As the Envoy image is large, select a layer that includes the binary instead of extracting all layers.
+  case ${version} in
+    1.19.*|1.20.*|1.21.*) # Envoy doesn't release old versions
+      ${car} --created-by-pattern 'ADD linux' --strip-components 2 -qf "${reference}" ${files}
+      ;;
+    *) # 1.22.0 changed the image layout.
+      ${car} --created-by-pattern '/usr/local/bin/envoy' --strip-components 2 -qf "${reference}" ${files}
+      ;;
+  esac
   ;;
 windows)
   reference=envoyproxy/envoy-windows:${version}

With the patch applied, the following should work for linux even if darwin is still waiting for someone to get homebrew working:

$ ./bin/archive_release_version.sh envoyproxy/envoy v1.21.0 check
$ ./bin/archive_release_version.sh envoyproxy/envoy v1.22.0 check

[codefromthecrypt]

Next action:

• someone raise a PR with the patch I mentioned and test it locally as mentioned
• once the above is merged, click "run workflow" here and enter the pattern as prompted. once for v1.22.0 and again for v1.22.0_debug

After that:

• someone try to get the homebrew 1.22.0 working
• once that is available on homebrew-core run the v1.22.0 release again.

Note: release steps are in the README, I'm just being specific.

[codefromthecrypt]

Tips on OS/x.

• Try local first as you can do this without effecting your laptop setup: brew install --build-from-source envoy --HEAD
• The actual syntax is something like brew bump-formula-pr --no-browse --no-audit --version v1.22.0
• Most Envoy minor releases break and you can enlist help once you have a PR up by pinging people who usually help. Ex https://github.com/Homebrew/homebrew-core/pull/90921

[codefromthecrypt]

here's a smaller diff that @cpakulski can use to work around this. I tested it.

diff --git a/bin/car_envoy.sh b/bin/car_envoy.sh
index ea1a34f..88a3176 100755
--- a/bin/car_envoy.sh
+++ b/bin/car_envoy.sh
@@ -110,7 +110,9 @@ linux)
     reference=envoyproxy/envoy:${version}
   fi

-  ${car} --created-by-pattern 'ADD linux' --strip-components 2 -qf "${reference}" ${files}
+  # Don't use --created-by-pattern because Envoy 1.22.0+ changed the image layer containing the binary.
+  # Note: Unlike windows, layers preceding the Envoy binary are small, so scanning through is OK.
+  ${car} --strip-components 2 -qf "${reference}" ${files}
   ;;
 windows)
   reference=envoyproxy/envoy-windows:${version}

$ ./bin/archive_release_version.sh envoyproxy/envoy v1.22.0 
archiving envoyproxy/envoy v1.22.0 released on 2022-04-15
using dist: envoy-v1.22.0-darwin-amd64
./bin/car_envoy.sh: line 98: tag: unbound variable
using dist: envoy-v1.22.0-darwin-arm64
./bin/car_envoy.sh: line 98: tag: unbound variable
using dist: envoy-v1.22.0-linux-amd64
creating envoy-v1.22.0-linux-amd64.tar.xz
using dist: envoy-v1.22.0-linux-arm64
creating envoy-v1.22.0-linux-arm64.tar.xz
using dist: envoy-v1.22.0-windows-amd64
creating envoy-v1.22.0-windows-amd64.tar.xz
using dist: envoy-v1.22.0-windows-arm64
error: windows/arm64 is not a supported platform: windows/amd64
$ tar -tvf v1.22.0//envoy-v1.22.0-linux-arm64.tar.xz
drwxr-xr-x  0 65534  65534       0 Apr 15 08:00 envoy-v1.22.0-linux-arm64/
drwxr-xr-x  0 65534  65534       0 Apr 15 08:00 envoy-v1.22.0-linux-arm64/bin/
-rwxr-xr-x  0 65534  65534 55335328 Apr 15 08:00 envoy-v1.22.0-linux-arm64/bin/envoy

Note: this only fixes the linux thing and only becomes usable when someone clicks release

[codefromthecrypt]

nit: let's avoid using the keyword "fixes" in PRs unless they do that as it closes the corresponding issue

NEXT STEPS:

• click through release workflow for v1.22.0 and v1.22.0_debug. The former should work for linux and windows and the latter linux only (as there's no debug release for windows or os/x)
• once homebrew has envoy 1.22.0 merged, re-click release v1.22.0 to add os/x to it. Also v1.21.2 once that's fixed which is probably the same issue.

cc also @jcchavezs as due to travel I may be sporadic and I think you can help guide @cpakulski as needed

[cpakulski]

Current status:

• linux and windows images are available for v1.22.0.
• linux image is available for v.22.0_debug

Next steps:

• fix homebrew for v1.22.0

[cpakulski]

Opened PR for homebrew-core to build version 1.21.2: https://github.com/Homebrew/homebrew-core/pull/101428

[cpakulski]

Current status:

macOS image obtained by car works only for macOS version 12. Envoy image fails when run on macOS 10.15 and 11. Next steps: Do not verify macOS image for macOS 10.15 and macOS 11.

[jcchavezs]

Are we stop supporting this versions or just disable verification. Also can you point out the failure or describe the error (for context purposes)

[cpakulski]

We stop supporting those. Verification fails because binary cannot be executed on versions 10.15 and 11. We would have to introduce another dimension to indicate OS version. However, users can install Envoy on 10.15 and 11 via brew, because it takes it from different bottles.

[jcchavezs]

Thanks for the clarification.

ons. 25. mai 2022, 20:41 skrev Christoph Pakulski @.***

:

We stop supporting those. Verification fails because binary cannot be executed on versions 10.15 and 11. We would have to introduce another dimension to indicate OS version. However, users can install Envoy on 10.15 and 11 via brew, because it takes it from different bottles.

— Reply to this email directly, view it on GitHub https://github.com/tetratelabs/archive-envoy/issues/34#issuecomment-1137705236, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYATD2UU3SQ7L2LBFDXTVLZX4BANCNFSM5UDW7TCA . You are receiving this because you were mentioned.Message ID: @.***>

[cpakulski]

Closing this. Nothing is left to be done for rel. 1.22.0.