How to Integrate Custom tcmalloc in Envoy Build for Jetson TX2 (Ubuntu 18.04, ARM64)

[yulinzou]

I'm encountering the issue #15235 with tcmalloc while I deploy kourier for knative, which uses envoy docker image, and I've identified the necessary changes to make in tcmalloc/internal/config.h for my Jetson TX2 platform (Ubuntu 18.04, ARM64). As Jetson TX2 takes a 3 level page table, and uses 39-bit virtual addresses.

My Challenge:

While I understand which part of tcmalloc to edit, I am unclear on how to ensure that Envoy uses this customized version of tcmalloc during its build process. Since tcmalloc is an external package required by Envoy, I need guidance on how to integrate my modified version into the Envoy build.

What I tried So Far:

Modifications to tcmalloc: I know which changes to make in tcmalloc/internal/config.h for ARM64 compatibility. Building Envoy: Ran ./ci/run_envoy_docker.sh './ci/do_ci.sh dev' to build the Envoy binary and have a rough idea of creating a Docker image based on this build.

What I Need Help With:

Integrating Custom tcmalloc: How can I modify the Envoy build process to use my customized tcmalloc? And once I have the Envoy binary built with the custom tcmalloc, what are the best practices for creating a Docker image from this build?

Any detailed steps, scripts, or pointers to relevant documentation would be greatly appreciated.

[adisuissa]

Generally speaking, Envoy is built using Bazel. And you may be able to modify the tcmalloc repo location here, and link your new package.

cc @KBaichoo @nezdolik who may have some ideas on custom tcmalloc usage.

[nezdolik]

+1 to what already has been mentioned, the least effort would be modifying the existing tcmalloc repository attributes and that should by default be compiled into Envoy dev binary. Depending on how you would like to include the tcmalloc binary, you may need yo use local native repository to reference that binary. And for building custom Envoy Docker image you could follow the guidelines here: https://www.envoyproxy.io/docs/envoy/latest/start/building/local_docker_build

[yulinzou]

Thank you for the suggestion.

I tried building directly with Bazel, but I encountered some configuration or dependency issues on my ARM device. I’m now considering using the script ./ci/run_envoy_docker.sh and passing the argument --define tcmalloc=gperftools.

I attempted to run:

./ci/run_envoy_docker.sh "BAZEL_BUILD_OPTIONS='--define tcmalloc=gperftools' ./ci/do_ci.sh dev"

However, this didn’t produce any output.

Could you please advise on the correct way to pass this argument?

[KBaichoo]

Try passing it instead in BAZEL_BUILD_EXTRA_OPTIONS which I think should modify the ci/build_setup.sh from do_ci.sh and source the BAZEL_BUILD_OPTIONS_LIST that the former script exports.