search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.02k stars 4.82k forks source link

Envoy 1.17-1.23 unable to start on Raspberry #23339

Closed Sitin closed 1 year ago

Sitin commented 2 years ago

Envoy crashes on RP 4 due to tcmalloc allocation error

I am trying to run Envoy on RP4 and 64bit Raspberry Pi OS.

Envoy 1.20 crashes with:

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x559342edc4 0x55934211a4 0x5593420be4 0x559340a4c0
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x559342f0d4 0x559340a534

Envoy 1.23 crashes with:

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:631] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x151880000000 1073741824 1073741824 @ 0x55941cfa88 0x55941cb6a0 0x55941caf48 0x55941b2f4c 0x55941c8010 0x55941c7e2c 0x55941a78a8 0x55940e6c00 0x55940e2298 0x559406f484 0x559419dbd0 0x7f834774ac
external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 600 @ 0x55941cfde4 0x55941b2fdc 0x55941c8010 0x55941c7e2c 0x55941a78a8 0x55940e6c00 0x55940e2298 0x559406f484 0x559419dbd0 0x7f834774ac

Tested also on envoy 1.17, 1.18, 1.19, and 1.22 (it seems that tcmalloc was introduced in 1.17). Same behaviour.

How to reproduce:

In Docker:

docker run --rm envoyproxy/envoy:v1.24-latest

On a host machine:

envoy_location=$(docker run --rm --entrypoint which docker.io/envoyproxy/envoy:v1.23-latest envoy)
container_id=$(docker create which envoyproxy/envoy:v1.23-latest)
docker cp "${container_id}":"${envoy_location}" ./envoy
docker rm -v "${container_id}"
./envoy

I am using RP4 8GB.

phlax commented 2 years ago

does it work if you just run with the container and not in the host environment?

Sitin commented 2 years ago

@phlax, both doesn't work for me.

I've actually tried docker, podman, and local run. Nothing works.

Sitin commented 2 years ago

I'll try to build with --define tcmalloc=disabled (I'll try containerised approach) and see whether it will work.

ggreenway commented 2 years ago

I'll try to build with --define tcmalloc=disabled (I'll try containerised approach) and see whether it will work.

tcmalloc=disabled is what I was going to suggest.

aadhi0319 commented 2 years ago

I just wanted to add that I have the same issue on a Raspberry Pi 3 (arm64). I need to run Envoy in a docker container. Is it possible to add the tmalloc=disabled to a docker container?

ggreenway commented 2 years ago

I just wanted to add that I have the same issue on a Raspberry Pi 3 (arm64). I need to run Envoy in a docker container. Is it possible to add the tmalloc=disabled to a docker container?

No, it's a compile-time option, so it requires a different build of envoy.

phlax commented 2 years ago

downstream issue https://github.com/raspberrypi/linux/issues/4375

phlax commented 2 years ago

seems like there is some resistance to switching the required kernel flags in raspberryPI OS - workarounds seem to be switching to debian or ubuntu or recompiling the kernel

PKizzle commented 2 years ago

Or compiling envoy with --define tcmalloc=gperftools which for me was the only acceptable option. I have published the images on Docker Hub https://hub.docker.com/repository/docker/thegrandpkizzle/envoy if you don't want to compile it yourself.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

febus982 commented 1 year ago

This is still present on 1.24

aadhi0319 commented 1 year ago

This is still present on 1.24

I put Ubuntu Lite on my RP4. Not ideal but it works well and Envoy works on it.

phlax commented 1 year ago

This is still present on 1.24

i think Envoy's tcmalloc requirements are unlikely to change (at least for published bins) so the resolution is to either recompile Envoy or use a kernel which supports it

nakermann1973 commented 8 months ago

Or compiling envoy with --define tcmalloc=gperftools which for me was the only acceptable option. I have published the images on Docker Hub https://hub.docker.com/repository/docker/thegrandpkizzle/envoy if you don't want to compile it yourself.

@PKizzle - is there any chance of you publishing updated images (or instructions for building suitable images myself)?

PKizzle commented 8 months ago

@nakermann1973 I have stopped using envoy due to the constant requirement of compiling the source code manually. Therefore, I cannot provide any new envoy images that use gperftools.

In order to build the images yourself you will need to run ./ci/run_envoy_docker.sh 'BAZEL_BUILD_EXTRA_OPTIONS="--define tcmalloc=gperftools" ./ci/do_ci.sh bazel.release.server_only' which will compile the envoy binary.

Then use the following Dockerfile (you need to update the dependencies) to build the alpine-based envoy image.

FROM alpine:3.16
RUN mkdir -p /etc/envoy

RUN apk --no-cache add ca-certificates wget
RUN wget -q -O /etc/apk/keys/devops.spectx.com.rsa.pub https://github.com/Lauri-Nomme/alpine-glibc-xb/releases/download/aarch64-2.33-r0/devops.spectx.com.rsa.pub \
    && wget https://github.com/Lauri-Nomme/alpine-glibc-xb/releases/download/aarch64-2.33-r0/glibc-2.33-r0.apk \
    && wget https://github.com/Lauri-Nomme/alpine-glibc-xb/releases/download/aarch64-2.33-r0/glibc-bin-2.33-r0.apk \
    && apk add glibc-2.33-r0.apk glibc-bin-2.33-r0.apk \
    && rm glibc-2.33-r0.apk glibc-bin-2.33-r0.apk

ADD configs/envoyproxy_io_proxy.yaml /etc/envoy/envoy.yaml
RUN apk add --no-cache shadow su-exec \
        && addgroup -S envoy && adduser --no-create-home -S envoy -G envoy

ARG ENVOY_BINARY_SUFFIX=_stripped
ADD linux/arm64/build_envoy_release${ENVOY_BINARY_SUFFIX}/* /usr/local/bin/

EXPOSE 10000

COPY ci/docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["envoy", "-c", "/etc/envoy/envoy.yaml"]

And this is the altered run_envoy_docker.sh script in order to compile envoy using nerdctl (containerd) instead of docker:

#!/bin/bash

set -e

# shellcheck source=ci/envoy_build_sha.sh
. "$(dirname "$0")"/envoy_build_sha.sh

function is_windows() {
  [[ "$(uname -s)" == *NT* ]]
}

read -ra ENVOY_DOCKER_OPTIONS <<< "${ENVOY_DOCKER_OPTIONS:-}"

# TODO(phlax): uppercase these env vars
export HTTP_PROXY="${http_proxy:-}"
export HTTPS_PROXY="${https_proxy:-}"
export NO_PROXY="${no_proxy:-}"
export GOPROXY="${go_proxy:-}"

if is_windows; then
  [[ -z "${IMAGE_NAME}" ]] && IMAGE_NAME="envoyproxy/envoy-build-windows2019"
  # TODO(sunjayBhatia): Currently ENVOY_DOCKER_OPTIONS is ignored on Windows because
  # CI sets it to a Linux-specific value. Undo this once https://github.com/envoyproxy/envoy/issues/13272
  # is resolved.
  ENVOY_DOCKER_OPTIONS=()
  # Replace MSYS style drive letter (/c/) with Windows drive letter designation (C:/)
  DEFAULT_ENVOY_DOCKER_BUILD_DIR=$(echo "${TEMP}" | sed -E "s#^/([a-zA-Z])/#\1:/#")/envoy-docker-build
  BUILD_DIR_MOUNT_DEST=C:/build
  SOURCE_DIR=$(echo "${PWD}" | sed -E "s#^/([a-zA-Z])/#\1:/#")
  SOURCE_DIR_MOUNT_DEST=C:/source
  START_COMMAND=("bash" "-c" "cd /c/source && export HOME=/c/build && $*")
else
  [[ -z "${IMAGE_NAME}" ]] && IMAGE_NAME="envoyproxy/envoy-build-ubuntu"
  # We run as root and later drop permissions. This is required to setup the USER
  # in useradd below, which is need for correct Python execution in the Docker
  # environment.
  ENVOY_DOCKER_OPTIONS+=(-u root:root)
  #ENVOY_DOCKER_OPTIONS+=(-v /var/run/docker.sock:/var/run/docker.sock)
  ENVOY_DOCKER_OPTIONS+=(--cap-add SYS_PTRACE --cap-add NET_RAW --cap-add NET_ADMIN)
  DEFAULT_ENVOY_DOCKER_BUILD_DIR=/tmp/envoy-docker-build
  BUILD_DIR_MOUNT_DEST=/build
  SOURCE_DIR="${PWD}"
  SOURCE_DIR_MOUNT_DEST=/source
  START_COMMAND=("/bin/bash" "-lc" "groupadd --gid $(id -g) -f envoygroup \
    && useradd -o --uid $(id -u) --gid $(id -g) --no-create-home --home-dir /build envoybuild \
    && usermod -a -G pcap envoybuild \
    && chown envoybuild:envoygroup /build \
    && sudo -EHs -u envoybuild bash -c 'cd /source && $*'")
fi

# The IMAGE_ID defaults to the CI hash but can be set to an arbitrary image ID (found with 'docker
# images').
[[ -z "${IMAGE_ID}" ]] && IMAGE_ID="${ENVOY_BUILD_SHA}"
[[ -z "${ENVOY_DOCKER_BUILD_DIR}" ]] && ENVOY_DOCKER_BUILD_DIR="${DEFAULT_ENVOY_DOCKER_BUILD_DIR}"
# Replace backslash with forward slash for Windows style paths
ENVOY_DOCKER_BUILD_DIR="${ENVOY_DOCKER_BUILD_DIR//\\//}"
mkdir -p "${ENVOY_DOCKER_BUILD_DIR}"

[[ -t 1 ]] && ENVOY_DOCKER_OPTIONS+=("-it")
[[ -f .git ]] && [[ ! -d .git ]] && ENVOY_DOCKER_OPTIONS+=(-v "$(git rev-parse --git-common-dir):$(git rev-parse --git-common-dir)")
[[ -n "${SSH_AUTH_SOCK}" ]] && ENVOY_DOCKER_OPTIONS+=(-v "${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK}" -e SSH_AUTH_SOCK)

export ENVOY_BUILD_IMAGE="${IMAGE_NAME}:${IMAGE_ID}"

VOLUMES=(
    -v "${ENVOY_DOCKER_BUILD_DIR}":"${BUILD_DIR_MOUNT_DEST}"
    -v "${SOURCE_DIR}":"${SOURCE_DIR_MOUNT_DEST}")

if ! is_windows; then
    # Create a "shared" directory that has the same path in/outside the container
    # This allows the host docker engine to see artefacts using a temporary path created inside the container,
    # at the same path.
    # For example, a directory created with `mktemp -d --tmpdir /tmp/bazel-shared` can be mounted as a volume
    # from within the build container.
    SHARED_TMP_DIR=/tmp/bazel-shared
    mkdir -p "${SHARED_TMP_DIR}"
    chmod +rwx "${SHARED_TMP_DIR}"
    VOLUMES+=(-v "${SHARED_TMP_DIR}":"${SHARED_TMP_DIR}")
fi

time nerdctl pull "${ENVOY_BUILD_IMAGE}"

# Since we specify an explicit hash, docker-run will pull from the remote repo if missing.
nerdctl run --rm \
       "${ENVOY_DOCKER_OPTIONS[@]}" \
       "${VOLUMES[@]}" \
       -e AZP_BRANCH \
       -e HTTP_PROXY \
       -e HTTPS_PROXY \
       -e NO_PROXY \
       -e GOPROXY \
       -e BAZEL_STARTUP_OPTIONS \
       -e BAZEL_BUILD_EXTRA_OPTIONS \
       -e BAZEL_EXTRA_TEST_OPTIONS \
       -e BAZEL_REMOTE_CACHE \
       -e ENVOY_STDLIB \
       -e BUILD_REASON \
       -e BAZEL_REMOTE_INSTANCE \
       -e GOOGLE_BES_PROJECT_ID \
       -e GCP_SERVICE_ACCOUNT_KEY \
       -e NUM_CPUS \
       -e ENVOY_RBE \
       -e ENVOY_BUILD_IMAGE \
       -e ENVOY_SRCDIR \
       -e ENVOY_BUILD_TARGET \
       -e ENVOY_BUILD_DEBUG_INFORMATION \
       -e SYSTEM_PULLREQUEST_PULLREQUESTNUMBER \
       -e GCS_ARTIFACT_BUCKET \
       -e GITHUB_TOKEN \
       -e BUILD_SOURCEBRANCHNAME \
       -e BAZELISK_BASE_URL \
       -e ENVOY_BUILD_ARCH \
       -e SLACK_TOKEN \
       -e BUILD_URI\
       -e REPO_URI \
       -e SYSTEM_STAGEDISPLAYNAME \
       -e SYSTEM_JOBDISPLAYNAME \
       -e SYSTEM_PULLREQUEST_PULLREQUESTNUMBER \
       "${ENVOY_BUILD_IMAGE}" \
       "${START_COMMAND[@]}"
FibreFoX commented 8 months ago

Just for sake of completeness, you can use a custom-compiled Raspberry Kernel with having CONFIG_PGTABLE_LEVELS changed:

# working when running as root
# install build requirements
apt install -y git bc bison flex libssl-dev make libncurses5-dev
# clone and download kernel sources
cd /opt
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux

# create kernel config
#RASPI 4: `make bcm2711_defconfig`
#RASPI 5: `make bcm2712_defconfig`

# # DO OTHER TUNINGS HERE via `make menuconfig`

# set PG table size to 4 instead of default 3
sed -i 's/CONFIG_PGTABLE_LEVELS=3/CONFIG_PGTABLE_LEVELS=4/' .config

# compile (requires a lot of time on raspi 4)
make -j4 Image.gz modules dtbs
make modules_install

# copy compiled resources
cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
cp arch/arm64/boot/dts/overlays/README /boot/overlays/
# RASPI 4: cp arch/arm64/boot/Image.gz /boot/kernel8.img
# RASPI 5: cp arch/arm64/boot/Image.gz /boot/firmware/kernel8.img

# use new kernel
echo "kernel=kernel8.img" | tee -a /boot/config.txt

reboot

This should make it working without a custom compiled envoy. Sadly the Raspberry issue https://github.com/raspberrypi/linux/issues/4375 (which is already mentioned in this thread https://github.com/envoyproxy/envoy/issues/23339#issuecomment-1267200594) is still open

I am running Hashicorp Nomad with that custom compiled kernel (with some other tweaks), hopefully works for others too.

j3ffrw commented 8 months ago

followed @FibreFoX instructions and the only change I did on it was on how to set CONFIG_PGTABLE_LEVELS as it gets reverted back to 3 once make gets executed.

cd /opt/linux
cat > .config-fragment << EOF
CONFIG_ARM64_VA_BITS_48=y
EOF
./scripts/kconfig/merge_config.sh .config .config-fragment
make -j4 Image.gz modules dtbs
# follow the rest
nakermann1973 commented 8 months ago

Thanks @FibreFoX @PKizzle .

I solved this a different way. I am now running Ubuntu 22.04 LTS on my Pi4's because its kernel appears to have the correct compilation options, and regular envoy docker images under nomad (consul connect) are running correctly.

$ uname -a
Linux clusterpi-03 5.15.0-1048-raspi #51-Ubuntu SMP PREEMPT Thu Feb 22 10:30:12 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
$ cat /boot/config* | grep CONFIG_PGTABLE_LEVELS
CONFIG_PGTABLE_LEVELS=4
CONFIG_PGTABLE_LEVELS=4
abasu0713 commented 5 months ago

Would really appreciate some help here. Been trying to deploy Envoy Gateway on Orange Pi 5B/Pro/Plus SBCs to no avail. 

alphaduriendur@hc-opi5b16-1:~$ cat /boot/config* | grep CONFIG_PGTABLE_LEVELS
CONFIG_PGTABLE_LEVELS=3
alphaduriendur@hc-opi5b16-1:~$ uname -a 
Linux hc-opi5b16-1.arkobasu.space 6.1.43-rockchip-rk3588 #1.0.8 SMP Wed Jun  5 13:38:18 CST 2024 aarch64 aarch64 aarch64 GNU/Linux
alphaduriendur@hc-opi5b16-1:~$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

alphaduriendur@hc-opi5b16-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:    22.04
Codename:   jammy
alphaduriendur@hc-opi5b16-1:~$
PKizzle commented 5 months ago

@abasu0713 Your CONFIG_PGTABLE_LEVELS must either be set to 4 by using a differently configured and compiled kernel (CONFIG_ARM64_VA_BITS_48=y) or by compiling envoy using --define tcmalloc=gperftools. That’s the only options for now. You might want to make the maintainers of the Orange Pi kernel aware of this issue and suggest changing the kernel configuration.

abasu0713 commented 5 months ago

@abasu0713 Your CONFIG_PGTABLE_LEVELS must either be set to 4 by using a differently configured and compiled kernel (CONFIG_ARM64_VA_BITS_48=y) or by compiling envoy using --define tcmalloc=gperftools. That’s the only options for now. You might want to make the maintainers of the Orange Pi kernel aware of this issue and suggest changing the kernel configuration.

I see I can update the Virtual Address space config in the kernel to 48 bit. The OS image is compiling. Will report back in a few hours after it completes and I test envoy on it. But kindly pardon my amateurish question @PKizzle - would updating this single kernel config have any negative impact on the image?

PKizzle commented 5 months ago

I see you already commented in https://github.com/raspberrypi/linux/issues/4375 where the implications of switching to 48 bit virtual address space are discussed for the kernel configuration used by Raspberry Pi OS. I am no kernel expert so I won’t try to add any more information to that. Ubuntu’s server version kernel configuration for their Raspberry Pi images switched to 48 bits quite a while ago so for any kind of negative impact you might want to check their bug tracker.

tristanmorgan commented 5 months ago

The alternative is for tcmalloc to be a little more flexible google/tcmalloc#82

abasu0713 commented 5 months ago

@abasu0713 Your CONFIG_PGTABLE_LEVELS must either be set to 4 by using a differently configured and compiled kernel (CONFIG_ARM64_VA_BITS_48=y) or by compiling envoy using --define tcmalloc=gperftools. That’s the only options for now. You might want to make the maintainers of the Orange Pi kernel aware of this issue and suggest changing the kernel configuration.

@PKizzle you are the absolute best! It works! I recompiled the kernel with CONFIG_ARM64_VA_BITS_48=y which was set to 39 bits as previously discussed. Strange that no one has updated it - I had the same issue using 3 different vendor ISOs:

Here's the logs:

OS Information:

  ___  ____  _   ____  ____  
 / _ \|  _ \(_) | ___|| __ ) 
| | | | |_) | | |___ \|  _ \ 
| |_| |  __/| |  ___) | |_) |
 \___/|_|   |_| |____/|____/ 

Welcome to Orange Pi 1.0.8 Bookworm with Linux 6.1.43-rockchip-rk3588

System load:   1%               Up time:       2 min    
Memory usage:  1% of 15.59G     IP:        
CPU temp:      30°C             Usage of /:    1% of 227G       

[ General system configuration (beta): orangepi-config ]

alphaduriendur@orangepi5b:~$ cat /boot/config* | grep CONFIG_PGTABLE_LEVELS
CONFIG_PGTABLE_LEVELS=4
alphaduriendur@orangepi5b:~$ uname -a
Linux orangepi5b 6.1.43-rockchip-rk3588 #1.0.8 SMP Thu Jun  6 06:38:04 CST 2024 aarch64 GNU/Linux
alphaduriendur@orangepi5b:~$ uname -r
6.1.43-rockchip-rk3588
alphaduriendur@orangepi5b:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Orange Pi 1.0.8 Bookworm
Release:    12
Codename:   bookworm

Installation with Kubernetes and metallb:

alphaduriendur@orangepi5b:~$ microk8s status --wait-ready
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
  disabled:
    cert-manager         # (core) Cloud native certificate management
    cis-hardening        # (core) Apply CIS K8s hardening
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    rook-ceph            # (core) Distributed Ceph storage using Rook
    storage              # (core) Alias to hostpath-storage add-on, deprecated
alphaduriendur@orangepi5b:~$ microk8s enable kube-ovn --force
Infer repository core for addon kube-ovn
Label node orangepi5b (192.168.5.44)
node/orangepi5b labeled
Remove Calico CNI
poddisruptionbudget.policy "calico-kube-controllers" deleted
serviceaccount "calico-kube-controllers" deleted
serviceaccount "calico-node" deleted
configmap "calico-config" deleted
customresourcedefinition.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "bgppeers.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "blockaffinities.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "caliconodestatuses.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "globalnetworksets.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamconfigs.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamhandles.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ippools.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipreservations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "kubecontrollersconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "networksets.crd.projectcalico.org" deleted
clusterrole.rbac.authorization.k8s.io "calico-kube-controllers" deleted
clusterrole.rbac.authorization.k8s.io "calico-node" deleted
clusterrolebinding.rbac.authorization.k8s.io "calico-kube-controllers" deleted
clusterrolebinding.rbac.authorization.k8s.io "calico-node" deleted
daemonset.apps "calico-node" deleted
deployment.apps "calico-kube-controllers" deleted
Deploy kube-ovn CRDs
customresourcedefinition.apiextensions.k8s.io/iptables-eips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-fip-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-dnat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-snat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/subnets.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vlans.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/provider-networks.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpcs.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpc-nat-gateways.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/security-groups.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/htbqoses.kubeovn.io created
Deploy ovn components
serviceaccount/ovn created
clusterrole.rbac.authorization.k8s.io/system:ovn created
clusterrolebinding.rbac.authorization.k8s.io/ovn created
service/ovn-nb created
service/ovn-sb created
service/ovn-northd created
deployment.apps/ovn-central created
daemonset.apps/ovs-ovn created
Deploy kube-ovn CNI
deployment.apps/kube-ovn-controller created
daemonset.apps/kube-ovn-cni created
daemonset.apps/kube-ovn-pinger created
deployment.apps/kube-ovn-monitor created
service/kube-ovn-monitor created
service/kube-ovn-pinger created
service/kube-ovn-controller created
service/kube-ovn-cni created
Install reconcile hook
alphaduriendur@orangepi5b:~$ alias kubectl="microk8s kubectl"
alphaduriendur@orangepi5b:~$ alias k8s="microk8s kubectl"
alphaduriendur@orangepi5b:~$ kubectl get po -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   coredns-864597b5fd-fp7wt               1/1     Running   0          10m
kube-system   kube-ovn-cni-9l4fl                     1/1     Running   0          47s
kube-system   kube-ovn-controller-8644466cbb-zlhv9   1/1     Running   0          2m23s
kube-system   kube-ovn-monitor-5485f75fc9-ch9jr      1/1     Running   0          2m23s
kube-system   kube-ovn-pinger-dlk2k                  1/1     Running   0          2m23s
kube-system   ovn-central-6f6cc5f7cc-frh88           1/1     Running   0          2m24s
kube-system   ovs-ovn-9hq88                          1/1     Running   0          2m24s
alphaduriendur@orangepi5b:~$ microk8s enable metallb:192.168.100.0/29
Infer repository core for addon metallb
Enabling MetalLB
Applying Metallb manifest
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
namespace/metallb-system created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
secret/webhook-server-cert created
service/webhook-service created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-configuration created
Waiting for Metallb controller to be ready.
deployment.apps/controller condition met
ipaddresspool.metallb.io/default-addresspool created
l2advertisement.metallb.io/default-advertise-all-pools created
MetalLB is enabled
alphaduriendur@orangepi5b:~$ kubectl get po -A
NAMESPACE        NAME                                   READY   STATUS    RESTARTS   AGE
kube-system      coredns-864597b5fd-fp7wt               1/1     Running   0          17m
kube-system      kube-ovn-cni-9l4fl                     1/1     Running   0          7m59s
kube-system      kube-ovn-controller-8644466cbb-zlhv9   1/1     Running   0          9m35s
kube-system      kube-ovn-monitor-5485f75fc9-ch9jr      1/1     Running   0          9m35s
kube-system      kube-ovn-pinger-dlk2k                  1/1     Running   0          9m35s
kube-system      ovn-central-6f6cc5f7cc-frh88           1/1     Running   0          9m36s
kube-system      ovs-ovn-9hq88                          1/1     Running   0          9m36s
metallb-system   controller-5f7bb57799-ksm48            1/1     Running   0          41s
metallb-system   speaker-lghxz                          1/1     Running   0          41s
alphaduriendur@orangepi5b:~$ uptime
 10:04:18 up 27 min,  1 user,  load average: 0.93, 1.09, 0.82

Installing Envoy gateway:

alphaduriendur@orangepi5b:~$ helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.0.1 -n envoy-gateway-system --create-namespace
NAME: eg
LAST DEPLOYED: Thu Jun  6 10:04:54 2024
NAMESPACE: envoy-gateway-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
**************************************************************************
*** PLEASE BE PATIENT: Envoy Gateway may take a few minutes to install ***
**************************************************************************

Envoy Gateway is an open source project for managing Envoy Proxy as a standalone or Kubernetes-based application gateway.

Thank you for installing Envoy Gateway! 🎉

Your release is named: eg. 🎉

Your release is in namespace: envoy-gateway-system. 🎉

To learn more about the release, try:

  $ helm status eg -n envoy-gateway-system
  $ helm get all eg -n envoy-gateway-system

To have a quickstart of Envoy Gateway, please refer to https://gateway.envoyproxy.io/latest/user/quickstart.

To get more details, please visit https://gateway.envoyproxy.io and https://github.com/envoyproxy/gateway.
alphaduriendur@orangepi5b:~$ kubectl get po -A
NAMESPACE              NAME                                   READY   STATUS    RESTARTS   AGE
envoy-gateway-system   envoy-gateway-57886575f6-s4z9q         1/1     Running   0          62s
kube-system            coredns-864597b5fd-fp7wt               1/1     Running   0          19m
kube-system            kube-ovn-cni-9l4fl                     1/1     Running   0          9m59s
kube-system            kube-ovn-controller-8644466cbb-zlhv9   1/1     Running   0          11m
kube-system            kube-ovn-monitor-5485f75fc9-ch9jr      1/1     Running   0          11m
kube-system            kube-ovn-pinger-dlk2k                  1/1     Running   0          11m
kube-system            ovn-central-6f6cc5f7cc-frh88           1/1     Running   0          11m
kube-system            ovs-ovn-9hq88                          1/1     Running   0          11m
metallb-system         controller-5f7bb57799-ksm48            1/1     Running   0          2m41s
metallb-system         speaker-lghxz                          1/1     Running   0          2m41s
alphaduriendur@orangepi5b:~$ kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
deployment.apps/envoy-gateway condition met
alphaduriendur@orangepi5b:~$ kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.0.1/quickstart.yaml -n default
gatewayclass.gateway.networking.k8s.io/eg created
gateway.gateway.networking.k8s.io/eg created
serviceaccount/backend created
service/backend created
deployment.apps/backend created
httproute.gateway.networking.k8s.io/backend created
alphaduriendur@orangepi5b:~$ kubectl get po -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                backend-96f75bbf-56s58                       1/1     Running   0          55s
envoy-gateway-system   envoy-default-eg-e41e7b31-59b4dd766f-hcgc2   2/2     Running   0          55s
envoy-gateway-system   envoy-gateway-57886575f6-s4z9q               1/1     Running   0          2m19s
kube-system            coredns-864597b5fd-fp7wt                     1/1     Running   0          20m
kube-system            kube-ovn-cni-9l4fl                           1/1     Running   0          11m
kube-system            kube-ovn-controller-8644466cbb-zlhv9         1/1     Running   0          12m
kube-system            kube-ovn-monitor-5485f75fc9-ch9jr            1/1     Running   0          12m
kube-system            kube-ovn-pinger-dlk2k                        1/1     Running   0          12m
kube-system            ovn-central-6f6cc5f7cc-frh88                 1/1     Running   0          12m
kube-system            ovs-ovn-9hq88                                1/1     Running   0          12m
metallb-system         controller-5f7bb57799-ksm48                  1/1     Running   0          3m58s
metallb-system         speaker-lghxz                                1/1     Running   0          3m58s
alphaduriendur@orangepi5b:~$ export GATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
error: arguments in resource/name form must have a single resource and name
alphaduriendur@orangepi5b:~$ export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
alphaduriendur@orangepi5b:~$ export GATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
alphaduriendur@orangepi5b:~$ curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
*   Trying 192.168.100.0:80...
* Connected to 192.168.100.0 (192.168.100.0) port 80 (#0)
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 06 Jun 2024 02:08:06 GMT
< content-length: 455
< 
{
 "path": "/get",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/7.88.1"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "192.168.5.44"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "a846099b-f8a8-4c18-b4f0-c18cf0f91d76"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "backend-96f75bbf-56s58"
* Connection #0 to host 192.168.100.0 left intact
alphaduriendur@orangepi5b:~$

Thanks again guys! This is why I love Envoy! Such an amazing product! and community! Cheers to you guys