search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.33k stars 682 forks source link

Not able to build FIPS complaint envoy for emissary ingress v3.9.1 #5470

Closed DeepakOhol closed 5 months ago

DeepakOhol commented 8 months ago

Describe the bug I am trying to build FIPS compliant envoy but not able to create it.

To Reproduce I have exported export FIPS_MODE=true and then used option make update-base to create FIPS envoy but getting following error

$ make update-base unset GIT_DIR GIT_WORK_TREE git init /home/nuance/3.9.1/emissary/_cxx/envoy cd /home/nuance/3.9.1/emissary/_cxx/envoy git remote get-url origin git remote set-url origin https://github.com/datawire/envoy.git [[ https://github.com/datawire/envoy.git == http://github.com/ ]] [[ https://github.com/datawire/envoy.git == https://github.com/ ]] git remote set-url --push origin git@github.com:datawire/envoy.git git fetch --tags origin '[' 6637fd1bab315774420f3c3d97488fedb7fc710f '!=' - ']' git checkout 6637fd1bab315774420f3c3d97488fedb7fc710f HEAD is now at 6637fd1bab response_map: fix refactoring that occurred between 1.26 - 1.27 pushd /home/nuance/3.9.1/emissary/_cxx/envoy/ci pwd echo /home/nuance/3.9.1/emissary/_cxx/envoy/ci . envoy_build_sha.sh uniq sed -e 's#.envoyproxy/envoy-build-ubuntu:(.)#\1#' dirname bash grep envoyproxy/envoy-build-ubuntu ./../.bazelrc ENVOY_BUILD_CONTAINER=fdd65c6270a8507a18d5acd6cf19a18cb695e4fa@sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e echo fdd65c6270a8507a18d5acd6cf19a18cb695e4fa@sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e cut -d@ -f1 ENVOY_BUILD_SHA=fdd65c6270a8507a18d5acd6cf19a18cb695e4fa echo fdd65c6270a8507a18d5acd6cf19a18cb695e4fa@sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e cut -d@ -f2 ENVOY_BUILD_CONTAINER_SHA=sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e [[ -n sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e ]] ENVOY_BUILD_CONTAINER_SHA=3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e awk '{$1=$1};1' wc -l [[ 1 == 1 ]] popd echo docker.io/envoyproxy/envoy-build-ubuntu:fdd65c6270a8507a18d5acd6cf19a18cb695e4fa tools/bin/write-ifchanged /home/nuance/3.9.1/emissary/_cxx/envoy-build-image.txt docker pull docker.io/emissaryingress/base-envoy:envoy-1.6637fd1bab315774420f3c3d97488fedb7fc710f.opt.FIPS Error response from daemon: manifest for emissaryingress/base-envoy:envoy-1.6637fd1bab315774420f3c3d97488fedb7fc710f.opt.FIPS not found: manifest unknown: manifest unknown unset GIT_DIR GIT_WORK_TREE git init /home/nuance/3.9.1/emissary/_cxx/envoy cd /home/nuance/3.9.1/emissary/_cxx/envoy git remote get-url origin git remote set-url origin https://github.com/datawire/envoy.git [[ https://github.com/datawire/envoy.git == http://github.com/ ]] [[ https://github.com/datawire/envoy.git == https://github.com/ ]] git remote set-url --push origin git@github.com:datawire/envoy.git git fetch --tags origin '[' 6637fd1bab315774420f3c3d97488fedb7fc710f '!=' - ']' git checkout 6637fd1bab315774420f3c3d97488fedb7fc710f HEAD is now at 6637fd1bab response_map: fix refactoring that occurred between 1.26 - 1.27 pushd /home/nuance/3.9.1/emissary/_cxx/envoy/ci pwd echo /home/nuance/3.9.1/emissary/_cxx/envoy/ci . envoy_build_sha.sh sed -e 's#.envoyproxy/envoy-build-ubuntu:(.)#\1#' dirname bash uniq grep envoyproxy/envoy-build-ubuntu ./../.bazelrc ENVOY_BUILD_CONTAINER=fdd65c6270a8507a18d5acd6cf19a18cb695e4fa@sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e cut -d@ -f1 echo fdd65c6270a8507a18d5acd6cf19a18cb695e4fa@sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e ENVOY_BUILD_SHA=fdd65c6270a8507a18d5acd6cf19a18cb695e4fa cut -d@ -f2 echo fdd65c6270a8507a18d5acd6cf19a18cb695e4fa@sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e ENVOY_BUILD_CONTAINER_SHA=sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e [[ -n sha256:3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e ]] ENVOY_BUILD_CONTAINER_SHA=3c8a3ce6f90dcfb5d09dc8f79bb01404d3526d420061f9a176e0a8e91e1e573e wc -l awk '{$1=$1};1' [[ 1 == 1 ]] popd tools/bin/write-ifchanged /home/nuance/3.9.1/emissary/_cxx/envoy-build-image.txt echo docker.io/envoyproxy/envoy-build-ubuntu:fdd65c6270a8507a18d5acd6cf19a18cb695e4fa tar: /home/nuance/3.9.1/emissary/_cxx/envoy-docker-build/envoy/x64/bin/release.tar.zst: Cannot open: No such file or directory tar: Error is not recoverable: exiting now cp: cannot stat ‘/home/nuance/3.9.1/emissary/_cxx/envoy-docker-build/envoy/x64/bin/dbg/envoy-contrib’: No such file or directory chmod: cannot access ‘/home/nuance/3.9.1/emissary/docker/base-envoy/envoy-static’: No such file or directory cp: cannot stat ‘/home/nuance/3.9.1/emissary/_cxx/envoy-docker-build/envoy/x64/bin/dbg/envoy-contrib.dwp’: No such file or directory chmod: cannot access ‘/home/nuance/3.9.1/emissary/docker/base-envoy/envoy-static.dwp’: No such file or directory cp: cannot stat ‘/home/nuance/3.9.1/emissary/_cxx/envoy-docker-build/envoy/x64/bin/envoy-contrib’: No such file or directory chmod: cannot access ‘/home/nuance/3.9.1/emissary/docker/base-envoy/envoy-static-stripped’: No such file or directory make[1]: ** [build-envoy] Error 1 make: [update-base] Error 2**

I have followed steps mentioned under https://github.com/emissary-ingress/emissary/pull/4279

Could you please help with correct way to build FIPS enabled envoy

cindymullins-dw commented 8 months ago

@DeepakOhol, it looks like an error with the tarball location or structure perhaps. @bathina2, firstly thanks for the contribution! Wonder if you can provide details on your successful FIPS build?

bathina2 commented 8 months ago

@cindymullins-dw You are right it does look like something has changed with the structure/location of the files. However, we are moving away from building emissary as it no longer fits our needs. We haven't attempted a successful build in a while.

shimpikk commented 8 months ago

@cindymullins-dw You are right it does look like something has changed with the structure/location of the files. However, we are moving away from building emissary as it no longer fits our needs. We haven't attempted a successful build in a while.

@bathina2 would you please elaborate on moving away from building emissary?

bathina2 commented 8 months ago

Our use case does not require something as feature-rich as emissary. Also building envoy in FIPS mode does not mean that the resulting emissary image is FIPS compliant. We have very strict requirements when it comes to FIPS and don't have the resources to fully qualify, maintain and build an emissary image in FIPS mode.

shimpikk commented 8 months ago

Hi, somehow "--define=boringssl=fips" flag is not getting supplied to envoy build via envoy.mk. So, we added it to BAZEL_GLOBAL_OPTIONS array in build_setup.sh. After that make update-based attempted the FIPS build of envoy, but it is failing with below error.

`+ /usr/local/bin/bazel --output_user_root=/build/bazel_root --output_base=/build/bazel_root/base build --repository_cache=/build/repository_cache --define=boringssl=fips --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --define --test_tmpdir=/build/tmp --config=libc++ --stripopt=--strip-all -c opt --remote_download_outputs=toplevel //distribution/binary:release Starting local Bazel server and connecting to it... INFO: Analyzed target //distribution/binary:release (1072 packages loaded, 83674 targets configured). INFO: Found 1 target... ERROR: /build/bazel_root/base/external/boringssl_fips/BUILD.bazel:25:8: Executing genrule @boringssl_fips//:build failed: (Exit 1): bash failed: error executing command (from target @boringssl_fips//:build) (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/40/execroot/envoy && \ exec env - \ BAZEL_COMPILER=clang \ BAZEL_CXXOPTS='-stdlib=libc++' \ BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \ BAZEL_LINKOPTS=-lm:-pthread \ CC=clang \ CXX=clang++ \ CXXFLAGS='-stdlib=libc++' \ LDFLAGS='-stdlib=libc++' \ LLVM_CONFIG=/opt/llvm/bin/llvm-config \ PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \ /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel/external/boringssl_fips.genrule_cmd bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a bazel-out/k8-opt/bin/external/boringssl_fips/ssl/libssl.a') Configuration: df220b683b93df8d53ce433a679550ae86b9467da48fcfdf93ae2ccabe11ad33 Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging dirname: missing operand Try 'dirname --help' for more information. pkgconfig not found. Disabling unwind tests. `

DeepakOhol commented 5 months ago

After running docker in privileged mode using following command docker run -itd --name testen --cap-add SYS_PTRACE --privileged -v $PWD/emissary:/home docker.io/envoyproxy/envoy-build-ubuntu:321658b6b50abda6869f89fac275f59bf3b1e757 bash and then running bazel --output_base=${WORKSPACE}/buildop build --define=boringssl=fips --repository_cache= --verbose_failures -c opt --config=clang //source/exe:envoy-static I was able to build FIPS complied envoy binary