Package pdf-to-img introduces bootstrap dependencies on ARM

[jbudz]

pdf-to-img was introduced as a way to test visual diffs on reports in https://github.com/elastic/kibana/pull/113910. This module depends on canvas https://github.com/Automattic/node-canvas, which does not have prebuilds for Darwin and Linux ARM. Building from source requires users and CI environments to install a number of dependencies - https://github.com/Automattic/node-canvas#compiling.

With Mac's trending towards ARM processors and our build pipeline using ARM instances to build our docker image, I'm hoping we can explore alternatives to reduce the amount of setup needed to run yarn kbn bootstrap

Truncated logs:

02:51:49 warning "pdf-to-img > pdfjs-dist@2.9.359" has unmet peer dependency "worker-loader@^3.0.7".
02:53:37 info Visit https://yarnpkg.com/en/docs/cli/remove for documentation about this command.
02:53:37 error /projects/kibana/node_modules/canvas: Command failed.
02:53:37 Exit code: 1
02:53:37 Command: node-pre-gyp install --fallback-to-build
02:53:37 Arguments: 
02:53:37 Directory: /projects/kibana/node_modules/canvas
02:53:37 Output:
02:53:37 node-pre-gyp info it worked if it ends with ok
02:53:37 node-pre-gyp info using node-pre-gyp@1.0.5
02:53:37 node-pre-gyp info using node@16.11.1 | linux | arm64
02:53:37 node-pre-gyp info check checked for "/projects/kibana/node_modules/canvas/build/Release/canvas.node" (not found)
02:53:37 node-pre-gyp http GET https://github.com/Automattic/node-canvas/releases/download/v2.8.0/canvas-v2.8.0-node-v93-linux-glibc-arm64.tar.gz
02:53:37 node-pre-gyp ERR! install response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.8.0/canvas-v2.8.0-node-v93-linux-glibc-arm64.tar.gz 
02:53:37 node-pre-gyp WARN Pre-built binaries not installable for canvas@2.8.0 and node@16.11.1 (node-v93 ABI, glibc) (falling back to source compile with node-gyp) 
02:53:37 node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.8.0/canvas-v2.8.0-node-v93-linux-glibc-arm64.tar.gz 
02:53:37 gyp info it worked if it ends with ok
02:53:37 gyp info using node-gyp@7.1.2
02:53:37 gyp info using node@16.11.1 | linux | arm64
02:53:37 gyp info ok 
02:53:37 gyp info it worked if it ends with ok
02:53:37 gyp info using node-gyp@7.1.2
02:53:37 gyp info using node@16.11.1 | linux | arm64
``

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[mistic]

@jbudz I believe one option is to contribute upstream and introduce the prebuildify on those dependencies missing it. Other option in case that is not possible is to replace that dependency with others which uses prebuildify builds.

[liza-mae]

@jbudz is this on bootstrap? I am noticing the following error also on bootstrap ubuntu arm.

6:04:38 [2021-10-26 22:04:38] [INFO] Installing node.js dependencies 
16:04:38 yarn run v1.22.17
16:04:38 $ node scripts/kbn bootstrap --prefer-offline
16:04:39  info [bazel_tools] installing Bazel tools
16:04:42  succ [bazel_tools] all bazel tools are correctly installed
16:04:42  info [bazel] Extracting Bazel installation...
16:04:58  info [bazel] Starting local Bazel server and connecting to it...
16:05:00  info [bazel] INFO: Invocation ID: 5c16f8af-8d09-43ba-970e-43d64b730a3b
16:05:05  info [bazel] INFO: Analyzed target @nodejs//:yarn (2 packages loaded, 3 targets configured).
16:05:05  info [bazel] INFO: Found 1 target...
16:05:06  info [bazel] INFO: Elapsed time: 23.888s, Critical Path: 0.23s
16:05:06  info [bazel] INFO: 1 process: 1 internal.
16:05:06  info [bazel] INFO: Running command line: /var/lib/jenkins/workspace/elastic+estf-kibana+7.16-snapshot+mj-linux-arm/JOB/basicGrp1/node/estf-arm-testing/.cache/bazel/_bazel_jenkins/8c2bb48ed7d1e488c80445c9ce14f569/external/nodejs_linux_arm64/bin/yarn
16:05:07  info [bazel] $ node ./preinstall_check
16:05:07  info [bazel] [1/5] Validating package.json...
16:05:07  info [bazel] [2/5] Resolving packages...
16:05:07  info [bazel] warning Resolution field "trim@1.0.1" is incompatible with requested version "trim@0.0.1"
16:05:07  info [bazel] warning Resolution field "trim@1.0.1" is incompatible with requested version "trim@0.0.1"
16:05:07  info [bazel] warning Resolution field "crypto-js@4.0.0" is incompatible with requested version "crypto-js@^3.1.9-1"
16:05:07  info [bazel] warning Resolution field "trim@1.0.1" is incompatible with requested version "trim@0.0.1"
16:05:07  info [bazel] warning Resolution field "typescript@4.1.3" is incompatible with requested version "typescript@^3.3.3333"
16:05:07  info [bazel] warning Resolution field "@types/node@16.10.2" is incompatible with requested version "@types/node@^10.1.0"
16:05:07  info [bazel] warning Resolution field "typescript@4.1.3" is incompatible with requested version "typescript@^4.3.5"
16:05:07  info [bazel] warning Resolution field "@types/node@16.10.2" is incompatible with requested version "@types/node@8.10.54"
16:05:07  info [bazel] warning Resolution field "typescript@4.1.3" is incompatible with requested version "typescript@~3.7.2"
16:05:08  info [bazel] warning Resolution field "@types/node@16.10.2" is incompatible with requested version "@types/node@^14.14.31"
16:05:08  info [bazel] warning Resolution field "typescript@4.1.3" is incompatible with requested version "typescript@^3.5.3"
16:05:08  info [bazel] warning Resolution field "schema-utils@1.0.0" is incompatible with requested version "schema-utils@^0.3.0"
16:05:08  info [bazel] warning Resolution field "trim@1.0.1" is incompatible with requested version "trim@0.0.1"
16:05:08  info [bazel] warning Resolution field "trim@1.0.1" is incompatible with requested version "trim@0.0.1"
16:05:08  info [bazel] [3/5] Fetching packages...
16:05:38  info [bazel] warning @elastic/synthetics@1.0.0-beta.13: Invalid bin entry for "@elastic/synthetics" (in "@elastic/synthetics").
16:05:46  info [bazel] warning sha.js@2.4.10: Invalid bin entry for "sha.js" (in "sha.js").
16:05:52  info [bazel] warning sha.js@2.4.11: Invalid bin entry for "sha.js" (in "sha.js").
16:06:04  info [bazel] info fsevents@2.1.2: The platform "linux" is incompatible with this module.
16:06:04  info [bazel] info "fsevents@2.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
16:06:04  info [bazel] [4/5] Linking dependencies...
16:06:04  info [bazel] warning " > @elastic/charts@38.0.1" has incorrect peer dependency "moment@^2.29.1".
16:06:04  info [bazel] warning " > @elastic/charts@38.0.1" has incorrect peer dependency "moment-timezone@^0.5.32".
16:06:04  info [bazel] warning " > @elastic/eui@39.1.1" has incorrect peer dependency "@elastic/datemath@^5.0.2".
16:06:04  info [bazel] warning " > @mapbox/mapbox-gl-rtl-text@0.2.3" has unmet peer dependency "mapbox-gl@>=0.32.1 <2.0.0".
16:06:04  info [bazel] warning "maplibre-gl > @mapbox/mapbox-gl-supported@1.5.0" has unmet peer dependency "mapbox-gl@>=0.32.1 <2.0.0".
16:06:04  info [bazel] warning " > ngreact@0.5.1" has unmet peer dependency "angular@< 2.0.0".
16:06:04  info [bazel] warning " > react-monaco-editor@0.41.2" has incorrect peer dependency "@types/react@^17.x".
16:06:04  info [bazel] warning " > react-monaco-editor@0.41.2" has incorrect peer dependency "react@^17.x".
16:06:04  info [bazel] warning " > react-shortcuts@2.0.1" has incorrect peer dependency "react@^0.14.8 || ^15".
16:06:04  info [bazel] warning " > react-shortcuts@2.0.1" has incorrect peer dependency "react-dom@^0.14.8 || ^15".
16:06:04  info [bazel] warning " > @elastic/eslint-plugin-eui@0.0.2" has incorrect peer dependency "eslint@>=5, <7".
16:06:04  info [bazel] warning "@octokit/rest > @octokit/plugin-request-log@1.0.2" has unmet peer dependency "@octokit/core@>=3".
16:06:04  info [bazel] warning "@storybook/addon-docs > @mdx-js/react@1.6.22" has incorrect peer dependency "react@^16.13.1 || ^17.0.0".
16:06:04  info [bazel] warning " > @storybook/testing-react@0.0.17" has unmet peer dependency "@storybook/client-api@^6 || >=6.0.0-0 || >=6.1.0-0 || >=6.2.0-0 || >=6.3.0-0 ".
16:06:04  info [bazel] warning "@testing-library/react-hooks > react-error-boundary@3.1.1" has incorrect peer dependency "react@>=16.13.1".
16:06:04  info [bazel] warning "pdf-to-img > pdfjs-dist@2.9.359" has unmet peer dependency "worker-loader@^3.0.7".
16:07:51  info [bazel] [5/5] Building fresh packages...
16:07:56  info [bazel] error /var/lib/jenkins/workspace/elastic+estf-kibana+7.16-snapshot+mj-linux-arm/JOB/basicGrp1/node/estf-arm-testing/kibana/node_modules/sharp: Command failed.
16:07:56  info [bazel] Exit code: 1
16:07:56  info [bazel] Command: (node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)
16:07:56  info [bazel] Arguments: 
16:07:56  info [bazel] Directory: /var/lib/jenkins/workspace/elastic+estf-kibana+7.16-snapshot+mj-linux-arm/JOB/basicGrp1/node/estf-arm-testing/kibana/node_modules/sharp
16:07:56  info [bazel] Output:
16:07:56  info [bazel] sharp: Installation error: Use with glibc 2.27 requires manual installation of libvips >= 8.10.6
16:07:56  info [bazel] sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies
16:07:56  info [bazel] info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
16:09:51 ERROR [bootstrap] failed:
16:09:51 ERROR The bazel command that was running failed to complete.
16:09:51 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
16:09:51 error Command failed with exit code 1.
16:09:51 [2021-10-26 22:09:51] [ERROR] yarn kbn bootstrap failed! 

[jbudz]

It's a similar issue, coming from elastic/synthetics -> sharp. We're somewhat lower priority on that one, prebuilds for darwin and Linux distros with glibc >= 2.29 are available.

[jloleysens]

Thanks for the details @jbudz; I can take a look at removing/changing this dependency.

[jloleysens]

I took a look at some alternatives to pdf-to-img, all of them require some kind of external dep (ImageMagick, GraphicsMagick, ghostscript, poppler - main ones I could see) I’m guessing this is not going to really be an improvement over the pre-built native code. Unless one of them is widely supported?

An alternative we could explore is to leverage the browser's canvas implementation. It'd involve some other trade-offs though:

• Adding a link to a CDN for pdfjs-dist or copying the lib and injecting it into the browser
• Passing a potentially long ArrayBuffer back and forth (not sure what the limits/issues here could be)

Not sure how feasible all of this is, but we would avoid relying on a pre-built in node_modules. I think the effort might not justify the benefits, though.

Otherwise the option mentioned by @mistic sounds promising.

I believe one option is to contribute upstream and introduce the prebuildify on those dependencies missing it

I'm happy to help out with that too!

CC @tsullivan

[tsullivan]

@jloleysens maybe we could decide to forego PDF comparison in CI and come up with some test scripts that should be run manually or in some other pipeline?

If Reporting didn't have export types that are PDF-only (Canvas and optimize-for-print in Dashboard), I would say we should stick with baseline comparison for PNG.

[jloleysens]

maybe we could decide to forego PDF comparison in CI and come up with some test scripts that should be run manually or in some other pipeline?

It would be unfortunate to lose this test coverage in CI, but if there is no relatively low-effort path to keeping it I think this sounds reasonable too.

[mistic]

@jloleysens is up to you guys to decide what to do here, but I think the best option is to remove the dependency from our pkg.json and disable the test for now, contribute upstream on https://github.com/Automattic/node-canvas to introduce https://github.com/prebuild/prebuildify so we dont need any local dependency to install that node module and then re-introduce the dep and the test again 😃

[jloleysens]

Closing this issue in favour of https://github.com/elastic/kibana/issues/117814 so Reporting team can track this work.

Feel free to re-open for Kibana operations' purposes 👍🏻