build [gcr.io/pixie-oss/pixie-dev/cloud/proxy_server_image] failed

[dragonTour]

Describe the bug As described in the [DEVELOPMENT.md]() documentation, an error occurred while proceeding to step skaffold dev -f skaffold/skaffold_cloud.yaml (-p dev) (-p minikube) (-p ory_auth).

Logs

...
Loaded image: bazel/src/cloud/profile:profile_server_image
Build [gcr.io/pixie-oss/pixie-dev/cloud/profile_server_image] succeeded
Building [gcr.io/pixie-oss/pixie-dev/cloud/proxy_server_image]...
Target platforms: [linux/amd64]
Loading: 
Loading: 0 packages loaded
INFO: Build options --action_env, --host_action_env, --host_linkopt, and 1 more have changed, discarding analysis cache.
Analyzing: target //src/cloud/proxy:proxy_prod_server_image.tar (1 packages loaded, 0 targets configured)
Analyzing: target //src/cloud/proxy:proxy_prod_server_image.tar (89 packages loaded, 1639 targets configured)
Analyzing: target //src/cloud/proxy:proxy_prod_server_image.tar (139 packages loaded, 10660 targets configured)
INFO: Analyzed target //src/cloud/proxy:proxy_prod_server_image.tar (145 packages loaded, 14513 targets configured).
INFO: Found 1 target...
[0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 1s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 12s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 32s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 42s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 62s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 92s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 106s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 139s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 160s processwrapper-sandbox
[104 / 117] Generating webpack deps src/ui/ui-deps.tar.gz; 209s processwrapper-sandbox
ERROR: /pl/src/px.dev/pixie/src/ui/BUILD.bazel:51:16: Generating webpack deps src/ui/ui-deps.tar.gz failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
Target //src/cloud/proxy:proxy_prod_server_image.tar failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 233.291s, Critical Path: 230.69s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

./px collect-logs

**App information (please complete the following information):**
- Pixie git tag:  release/cloud/prod/1667424853

[ddelnano]

Hi @dragonTour, can you run the following bazel command and provide the output?

bazel build //src/cloud/proxy:all

It's hard to tell why the bazel command failed when skaffold is truncating the output (as indicated from the line below)

ERROR: /pl/src/px.dev/pixie/src/ui/BUILD.bazel:51:16: Generating webpack deps src/ui/ui-deps.tar.gz failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)

[dragonTour]

@ddelnano Thank you for your reply，Executing the bazel command bazel build //src/cloud/proxy:all yields the same error message as above.

parallels@px-dev-docker:/pl/src/px.dev/pixie$ bazel build //src/cloud/proxy:all
INFO: Analyzed 8 targets (0 packages loaded, 0 targets configured).
INFO: Found 8 targets...
ERROR: /pl/src/px.dev/pixie/src/ui/BUILD.bazel:51:16: Generating webpack deps src/ui/ui-deps.tar.gz failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 522.871s, Critical Path: 522.02s
INFO: 4 processes: 3 internal, 1 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

[ddelnano]

@dragonTour I mistyped my command. Please rerun that with —sandbox_debug. It will be helpful to see the entire error log.

[dragonTour]

@ddelnano The following is the error log

parallels@px-dev-docker:/pl/src/px.dev/pixie$ bazel build --sandbox_debug //src/cloud/proxy:all                
INFO: Analyzed 8 targets (0 packages loaded, 0 targets configured).
INFO: Found 8 targets...
ERROR: /pl/src/px.dev/pixie/src/ui/BUILD.bazel:51:16: Generating webpack deps src/ui/ui-deps.tar.gz failed: (Exit 1): process-wrapper failed: error executing command 
  (cd /home/parallels/.cache/bazel/_bazel_parallels/4c31fb537ca0f31ab15bbd6a8445d3b6/sandbox/processwrapper-sandbox/9/execroot/px && \
  exec env - \
    BASE_PATH='$$PWD' \
    TMPDIR=/tmp \
    UILIB_PATH=src/ui \
  /home/parallels/.cache/bazel/_bazel_parallels/install/cbf972266931ad9fad1857441b832915/process-wrapper '--timeout=0' '--kill_delay=15' /bin/bash -c 'export BASE_PATH=$PWD && export PATH=/usr/local/bin:/opt/node/bin:$PATH && export HOME=$(mktemp -d) && export TMPPATH=$(mktemp -d) && if [ -f "bazel-out/stable-status.txt" ]; then
$(sed -E "s/^([A-Za-z_]+)\s*(.*)/export \1=\2/g" bazel-out/stable-status.txt)
fi && if [ -f "bazel-out/volatile-status.txt" ]; then
$(sed -E "s/^([A-Za-z_]+)\s*(.*)/export \1=\2/g" bazel-out/volatile-status.txt)
fi && cp -aL ${BASE_PATH}/* ${TMPPATH} && pushd ${TMPPATH}/${UILIB_PATH} &> /dev/null && export OUTPUT_PATH=bazel-out/k8-fastbuild/bin/src/ui/ui-deps.tar.gz && yarn install --immutable &> build.log && tar -czf ${BASE_PATH}/${OUTPUT_PATH} . && popd &> /dev/null && rm -rf ${TMPPATH}')
INFO: Elapsed time: 869.608s, Critical Path: 868.45s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully

[ddelnano]

@dragonTour how did you set up your Pixie development environment? Are you running this command within our development docker image or did you run the chef against your development machine?

If not, I would start there because I suspect your development dependencies aren't at the correct versions.

[dragonTour]

I run his command within our development docker image.

Now, I go straight to src/ui and execute the yum install command，Will prompt information Some peer dependencies are incorrectly met

parallels@px-dev-docker:/pl/src/px.dev/pixie/src/ui$ yarn install
➤ YN0000: ┌ Resolution step
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide @egjs/hammerjs (p122f6), requested by vis-network
➤ YN0060: │ @pixie-labs/ui@workspace:. provides @types/react (pa2d32) with version 18.0.9, which doesn't satisfy what @mui/system and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides @types/react (pb2f4f) with version 18.0.9, which doesn't satisfy what @mui/icons-material requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides @types/react (p32884) with version 18.0.9, which doesn't satisfy what @mui/material and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides @types/react (p92011) with version 18.0.9, which doesn't satisfy what @mui/styles and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides @types/react (pb390d) with version 18.0.9, which doesn't satisfy what @testing-library/react-hooks and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides @types/react (p70861) with version 18.0.9, which doesn't satisfy what react-hot-loader requests
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide component-emitter (pf15ae), requested by vis-network
➤ YN0060: │ @pixie-labs/ui@workspace:. provides graphql (p488bf) with version 15.5.0, which doesn't satisfy what graphql-schema-typescript requests
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide keycharm (pffea8), requested by vis-network
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide moment (p6c62b), requested by vis-network
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p2a764) with version 18.1.0, which doesn't satisfy what @mui/system and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p7f1f3) with version 18.1.0, which doesn't satisfy what react-router and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p6cd17) with version 18.1.0, which doesn't satisfy what @apollo/client requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p21644) with version 18.1.0, which doesn't satisfy what @mui/icons-material requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p90eb3) with version 18.1.0, which doesn't satisfy what @mui/material and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (pf62fe) with version 18.1.0, which doesn't satisfy what @mui/styles and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p87a7f) with version 18.1.0, which doesn't satisfy what @mui/utils requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p0a807) with version 18.1.0, which doesn't satisfy what @testing-library/react-hooks and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (pba865) with version 18.1.0, which doesn't satisfy what launchdarkly-react-client-sdk requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p0cdf4) with version 18.1.0, which doesn't satisfy what react-cookie-banner and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p26c94) with version 18.1.0, which doesn't satisfy what react-hot-loader requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (pbc6c2) with version 18.1.0, which doesn't satisfy what react-monaco-editor requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p0c41b) with version 18.1.0, which doesn't satisfy what react-resizable and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (p410b9) with version 18.1.0, which doesn't satisfy what react-table requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (pc5faf) with version 18.1.0, which doesn't satisfy what react-vega requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (pc450e) with version 18.1.0, which doesn't satisfy what react-virtualized-auto-sizer requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react (pc2fd0) with version 18.1.0, which doesn't satisfy what react-window requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (p97db7) with version 18.1.0, which doesn't satisfy what @mui/material and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (p173d3) with version 18.1.0, which doesn't satisfy what @testing-library/react-hooks requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (p82f08) with version 18.1.0, which doesn't satisfy what launchdarkly-react-client-sdk requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (p46ab3) with version 18.1.0, which doesn't satisfy what react-hot-loader requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (p98e64) with version 18.1.0, which doesn't satisfy what react-resizable and some of its descendants request
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (pf9c39) with version 18.1.0, which doesn't satisfy what react-virtualized-auto-sizer requests
➤ YN0060: │ @pixie-labs/ui@workspace:. provides react-dom (pb15d2) with version 18.1.0, which doesn't satisfy what react-window requests
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide timsort (p92522), requested by vis-network
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide uuid (p17ed1), requested by vis-network
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide vis-data (p08299), requested by vis-network
➤ YN0002: │ @pixie-labs/ui@workspace:. doesn't provide vis-util (p6e107), requested by vis-network
➤ YN0060: │ @pixie-labs/ui@workspace:. provides webpack (peb836) with version 5.55.0, which doesn't satisfy what monaco-editor-webpack-plugin requests
➤ YN0002: │ eslint-config-airbnb-typescript@npm:16.1.0 [6311d] doesn't provide eslint (p06d79), requested by eslint-config-airbnb-base
➤ YN0002: │ eslint-config-airbnb-typescript@npm:16.1.0 [6311d] doesn't provide eslint-plugin-import (p38fd9), requested by eslint-config-airbnb-base
➤ YN0002: │ react-grid-layout@npm:0.18.3 doesn't provide react (p9964f), requested by react-draggable
➤ YN0002: │ react-grid-layout@npm:0.18.3 doesn't provide react (p08b23), requested by react-resizable
➤ YN0002: │ react-grid-layout@npm:0.18.3 doesn't provide react-dom (p8ffc4), requested by react-draggable
➤ YN0002: │ react-grid-layout@npm:0.18.3 doesn't provide react-dom (p0b1cc), requested by react-resizable
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 0s 270ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 2s 484ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0000: └ Completed in 0s 423ms
➤ YN0000: Done with warnings in 3s 621ms

[dragonTour]

run bazel yarn install --immutable &>build.log,and get the error

➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ canvas@npm:2.8.0 must be built because it never has been before or the last one failed
➤ YN0007: │ cypress@npm:9.3.0 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.13.13 must be built because it never has been before or the last one failed
➤ YN0007: │ sharp@npm:0.30.5 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:3.13.1 must be built because it never has been before or the last one failed
➤ YN0007: │ vis-network@npm:7.10.2 [6311d] must be built because it never has been before or the last one failed
➤ YN0009: │ sharp@npm:0.30.5 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-00bac834/build.log)
➤ YN0009: │ canvas@npm:2.8.0 couldn't be built successfully (exit code 6, logs can be found here: /tmp/xfs-3f26c880/build.log)
➤ YN0000: └ Completed in 27m 27s
➤ YN0000: Failed with errors in 29m 15s

in file /tmp/xfs-00bac834/build.log

# This file contains the result of Yarn building a package (sharp@npm:0.30.5)
# Script name: install

sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.12.2/libvips-8.12.2-linux-x64.tar.br
sharp: Installation error: connect ECONNREFUSED 20.205.243.166:443
sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies

and in file /tmp/xfs-3f26c880/build.log

# This file contains the result of Yarn building a package (canvas@npm:2.8.0)
# Script name: install

node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@1.0.6
node-pre-gyp info using node@16.13.2 | linux | x64
node-pre-gyp info check checked for "/tmp/tmp.JpKtQ8oZMJ/src/ui/.yarn/unplugged/canvas-npm-2.8.0-b8a304d4de/node_modules/canvas/build/Release/canvas.node" (not found)
node-pre-gyp http GET https://github.com/Automattic/node-canvas/releases/download/v2.8.0/canvas-v2.8.0-node-v93-linux-glibc-x64.tar.gz
node-pre-gyp info install unpacking Release/
node-pre-gyp info install unpacking Release/libgmodule-2.0.so.0
node-pre-gyp info install unpacking Release/libz.so.1
node-pre-gyp info install unpacking Release/libcairo.so.2
node-pre-gyp info install unpacking Release/libgdk_pixbuf-2.0.so.0
node-pre-gyp info install unpacking Release/libpixman-1.so.0
node-pre-gyp info install unpacking Release/libffi.so.7
node-pre-gyp info install unpacking Release/libxml2.so.2
node-pre-gyp ERR! Completion callback never invoked! 
node-pre-gyp ERR! System Linux 6.2.0-31-generic
node-pre-gyp ERR! command "/opt/node/bin/node" "/tmp/tmp.JpKtQ8oZMJ/src/ui/.yarn/cache/@mapbox-node-pre-gyp-npm-1.0.6-5055e7dcf5-1ea1089e01.zip/node_modules/@mapbox/node-pre-gyp/bin/node-pre-gyp" "install" "--fa
llback-to-build"
node-pre-gyp ERR! cwd /tmp/tmp.JpKtQ8oZMJ/src/ui/.yarn/unplugged/canvas-npm-2.8.0-b8a304d4de/node_modules/canvas
node-pre-gyp ERR! node -v v16.13.2
node-pre-gyp ERR! node-pre-gyp -v v1.0.6

[ddelnano]

@dragonTour the bazel build process should be using the node bundled within the docker image, which is at v18.16.0.

$ ./scripts/run_docker.sh
$ which node
/opt/px_dev/tools/node/bin/node
$ /opt/px_dev/tools/node/bin/node --version
v18.16.0

How was the node at /opt/node/bin/node installed?

[dragonTour]

The docker image dosen't have this path I/opt/px_dev/tools/node/bin/node, it has ownself node in /opt/node/bin/node,

the version is

$ cd opt/node/bin
$ ./node -v
$ v16.13.2

[ddelnano]

There should not be an /opt/node. The only top level directories within /opt should be px_dev and google-cloud-sdk

# This uses the latest px_dev_image tag
$ docker create px_dev_image:202306170607
803152ffde89dd8c2859669e2fa8e7f951aa34b63ed3261b73615f7f8df17085
$ docker cp 803152ffde89dd8c2859669e2fa8e7f951aa34b63ed3261b73615f7f8df17085:/opt ./
$ ls -alh opt/
total 24K
drwxr-xr-x  4 ddelnano ddelnano 4.0K Jun 17 06:17 .
drwxrwxr-x 20 ddelnano ddelnano  12K Sep  7 14:11 ..
drwxr-xr-x  9 ddelnano ddelnano 4.0K Jun 17 06:16 google-cloud-sdk
drwxr-xr-x  5 ddelnano ddelnano 4.0K Jun 17 06:14 px_dev

What docker image and tag are you using? Please provide the fully qualified image name and sha256 for it.

[dragonTour]

$ docker images -a --digests
REPOSITORY              TAG               DIGEST                    IMAGE ID    
px_dev_image            202210241510     <none>                     12b130327a7a   

$ docker inspect 12b130327a7a
[
    {
        "Id": "sha256:12b130327a7aa0983c56e4253737a5b8ff20a0a23778f3cc7c08f042ee39c4ac",
        "RepoTags": [
            "px_dev_image:202210241510"
        ],
       .....
    }
]

The tag 202210241510 is defined in docker.properties。 The git branch is below:

$ git branch
$* (HEAD detached at release/cloud/prod/1667424853)
   main

Line 24 of the filepixie/bazel/ui.bazl ，export PATH=/usr/local/bin:/opt/node/bin:$PATH, so mybe can't change the node path

[ddelnano]

@dragonTour unfortunately that commit is quite old and we don't provide any guarantees for supporting old builds from source. I'm going to close this issue because this problem isn't reproducible on a recent commit (if this happens on main please let me know but I think it's unlikely to manifest in this way).

Is there a reason you are trying to develop against a cloud release from last year? When developing you should always branch off a recent main commit and rebase if the branch will be long lived.

[dragonTour]

It's because the newer image versions cann't run pixie_cloud or viser in my own kubernates environment, or had web UI login issues. I will try to use the latest release.