search

elastic / apm-integration-testing

APM Integration Testing
Apache License 2.0
54 stars 68 forks source link

Failed to download Chromium r686378 when running the all.sh #851

Closed v1v closed 4 years ago

v1v commented 4 years ago

build failed as a consequence of the downstream build, see the below stacktrace:

[2020-06-05T03:14:50.506Z] + .ci/scripts/all.sh
[2020-06-05T03:14:50.506Z] ELASTIC_STACK_VERSION=8.0.0
[2020-06-05T03:14:50.506Z] APM_SERVER_BRANCH_VERSION=
[2020-06-05T03:14:50.506Z] APM_SERVER_BRANCH_TYPE=
[2020-06-05T03:14:50.506Z] BUILD_OPTS=  --build-parallel
[2020-06-05T03:14:50.506Z] test -d /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976 || virtualenv -q --python=python3 /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976;\
[2020-06-05T03:14:50.506Z] source /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976/bin/activate || exit 1;\
[2020-06-05T03:14:50.506Z] pip install -q -r requirements.txt;\
[2020-06-05T03:14:50.506Z] touch /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976;\
[2020-06-05T03:14:50.506Z] 
[2020-06-05T03:15:08.603Z] make start-env
[2020-06-05T03:15:08.603Z] make[1]: Entering directory '/var/lib/jenkins/workspace/tegration-test-downstream_master/src/github.com/elastic/apm-integration-testing'
[2020-06-05T03:15:08.603Z] test -d /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976 || virtualenv -q --python=python3 /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976;\
[2020-06-05T03:15:08.603Z] source /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976/bin/activate || exit 1;\
[2020-06-05T03:15:08.603Z] pip install -q -r requirements.txt;\
[2020-06-05T03:15:08.603Z] touch /var/lib/jenkins/workspace/tegration-test-downstream_mastervenv-2976;\
[2020-06-05T03:15:08.603Z] 
[2020-06-05T03:15:08.603Z] python3 scripts/compose.py start 8.0.0   --build-parallel  --no-apm-server-dashboards   --no-apm-server-self-instrument   --with-agent-rumjs   --with-agent-dotnet   --with-agent-go-net-http   --with-agent-nodejs-express   --with-agent-ruby-rails   --with-agent-java-spring   --with-agent-python-django   --with-agent-python-flask   --force-build   --no-xpack-secure   --apm-server-enable-tls   --no-verify-server-cert    --apm-server-secret-token=SuPeRsEcReT   --apm-server-url=https://apm-server:8200
...
...
[2020-06-05T03:18:56.283Z] husky > Setting up git hooks
[2020-06-05T03:18:56.283Z] ENOENT: no such file or directory, mkdir 'node_modules/husky/.git/hooks'
[2020-06-05T03:18:56.283Z] husky > Failed to install
[2020-06-05T03:18:56.283Z] 
[2020-06-05T03:18:56.283Z] > playwright@0.11.1 install /rumjs-integration-test/node_modules/playwright
[2020-06-05T03:18:56.283Z] > node install.js
[2020-06-05T03:18:56.283Z] 
[2020-06-05T03:18:59.094Z] 
[2020-06-05T03:19:08.474Z] chromium downloaded to /rumjs-integration-test/node_modules/playwright-core/.local-chromium/linux-740847
[2020-06-05T03:19:09.416Z] 
[2020-06-05T03:19:11.967Z] firefox downloaded to /rumjs-integration-test/node_modules/playwright-core/.local-firefox/linux-1029
[2020-06-05T03:20:48.464Z] 
[2020-06-05T03:20:48.464Z] webkit downloaded to /rumjs-integration-test/node_modules/playwright-core/.local-webkit/linux-1151
[2020-06-05T03:20:48.464Z] 
[2020-06-05T03:20:48.464Z] > puppeteer@1.20.0 install /rumjs-integration-test/node_modules/puppeteer
[2020-06-05T03:20:48.464Z] > node install.js
[2020-06-05T03:20:48.464Z] 
[2020-06-05T03:20:48.464Z] ERROR: Failed to download Chromium r686378! Set "PUPPETEER_SKIP_CHROMIUM_DOWNLOAD" env variable to skip download.
[2020-06-05T03:20:48.464Z] Error: read ECONNRESET
[2020-06-05T03:20:48.464Z]     at TLSWrap.onStreamRead (internal/stream_base_commons.js:205:27)
[2020-06-05T03:20:48.464Z]   -- ASYNC --
[2020-06-05T03:20:48.464Z]     at BrowserFetcher.<anonymous> (/rumjs-integration-test/node_modules/puppeteer/lib/helper.js:111:15)
[2020-06-05T03:20:48.464Z]     at Object.<anonymous> (/rumjs-integration-test/node_modules/puppeteer/install.js:64:16)
[2020-06-05T03:20:48.464Z]     at Module._compile (internal/modules/cjs/loader.js:1138:30)
[2020-06-05T03:20:48.464Z]     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
[2020-06-05T03:20:48.464Z]     at Module.load (internal/modules/cjs/loader.js:986:32)
[2020-06-05T03:20:48.464Z]     at Function.Module._load (internal/modules/cjs/loader.js:879:14)
[2020-06-05T03:20:48.464Z]     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
[2020-06-05T03:20:48.464Z]     at internal/main/run_main_module.js:17:47 {
[2020-06-05T03:20:48.464Z]   errno: 'ECONNRESET',
[2020-06-05T03:20:48.464Z]   code: 'ECONNRESET',
[2020-06-05T03:20:48.464Z]   syscall: 'read'
[2020-06-05T03:20:48.464Z] }
[2020-06-05T03:20:48.725Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
[2020-06-05T03:20:48.725Z] npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[2020-06-05T03:20:48.725Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.2 (node_modules/karma/node_modules/fsevents):
[2020-06-05T03:20:48.726Z] npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[2020-06-05T03:20:48.726Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.2 (node_modules/@wdio/cli/node_modules/fsevents):
[2020-06-05T03:20:48.726Z] npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[2020-06-05T03:20:48.726Z] 
[2020-06-05T03:20:48.726Z] npm ERR! code ELIFECYCLE
[2020-06-05T03:20:48.727Z] npm ERR! errno 1
[2020-06-05T03:20:48.727Z] npm ERR! puppeteer@1.20.0 install: `node install.js`
[2020-06-05T03:20:48.727Z] npm ERR! Exit status 1
[2020-06-05T03:20:48.727Z] npm ERR! 
[2020-06-05T03:20:48.727Z] npm ERR! Failed at the puppeteer@1.20.0 install script.
[2020-06-05T03:20:48.727Z] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[2020-06-05T03:20:48.727Z] 
[2020-06-05T03:20:48.727Z] npm ERR! A complete log of this run can be found in:
[2020-06-05T03:20:48.727Z] npm ERR!     /root/.npm/_logs/2020-06-05T03_20_48_557Z-debug.log
...
...
[2020-06-05T03:23:19.499Z] ERROR: for agent-rumjs  (<Service: agent-rumjs>, "The command '/bin/sh -c npm install' returned a non-zero code: 1")
[2020-06-05T03:23:19.499Z] Service 'agent-rumjs' failed to build: The command '/bin/sh -c npm install' returned a non-zero code: 1
[2020-06-05T03:23:19.499Z] Starting stack services..
[2020-06-05T03:23:19.499Z] 
[2020-06-05T03:23:19.499Z] Traceback (most recent call last):
[2020-06-05T03:23:19.499Z]   File "scripts/compose.py", line 25, in <module>
[2020-06-05T03:23:19.499Z]     main()
[2020-06-05T03:23:19.499Z]   File "scripts/compose.py", line 21, in main
[2020-06-05T03:23:19.499Z]     setup()
[2020-06-05T03:23:19.499Z]   File "/var/lib/jenkins/workspace/tegration-test-downstream_master/src/github.com/elastic/apm-integration-testing/scripts/modules/cli.py", line 178, in __call__
[2020-06-05T03:23:19.499Z]     self.args.func()
[2020-06-05T03:23:19.499Z]   File "/var/lib/jenkins/workspace/tegration-test-downstream_master/src/github.com/elastic/apm-integration-testing/scripts/modules/cli.py", line 618, in start_handler
[2020-06-05T03:23:19.499Z]     self.run_docker_compose_process(docker_compose_build + build_services)
[2020-06-05T03:23:19.499Z]   File "/var/lib/jenkins/workspace/tegration-test-downstream_master/src/github.com/elastic/apm-integration-testing/scripts/modules/cli.py", line 405, in run_docker_compose_process
[2020-06-05T03:23:19.499Z]     subprocess.check_call(docker_compose_cmd)
[2020-06-05T03:23:19.499Z]   File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
[2020-06-05T03:23:19.499Z]     raise CalledProcessError(retcode, cmd)
[2020-06-05T03:23:19.499Z] subprocess.CalledProcessError: Command '['docker-compose', '-f', '/var/lib/jenkins/workspace/tegration-test-downstream_master/src/github.com/elastic/apm-integration-testing/docker-compose.yml', '--no-ansi', '--log-level', 'ERROR', 'build', '--pull', '--no-cache', '--parallel', 'agent-rumjs', 'agent-python-django', 'agent-ruby-rails', 'agent-java-spring', 'agent-dotnet', 'agent-go-net-http', 'agent-python-flask', 'agent-nodejs-express']' returned non-zero exit status 1.
[2020-06-05T03:23:19.499Z] Makefile:62: recipe for target 'start-env' failed
[2020-06-05T03:23:19.499Z] make[1]: *** [start-env] Error 1
[2020-06-05T03:23:19.499Z] make[1]: Leaving directory '/var/lib/jenkins/workspace/tegration-test-downstream_master/src/github.com/elastic/apm-integration-testing'
[2020-06-05T03:23:19.499Z] Makefile:73: recipe for target 'env-agent-all' failed
[2020-06-05T03:23:19.499Z] make: *** [env-agent-all] Error 2
script returned exit code 2

There was a kind of fix back in the days for the opbeans-frontend:

But it seems the issue still persists. Besides @vigneshshanmugam pointed the env variable might not be the way to fix it. @kuisathaverat suggested to use our well known retry/sleep framework.

I wonder if the above fix is valid though, there is a comment regarding that particular env variable:

@vigneshshanmugam and @jahtalab would you mind to have a look?

hmdhk commented 4 years ago

@v1v Is it possible to mirror the chrome download artefacts within our infra? there's an env variable PUPPETEER_DOWNLOAD_HOST that we can use to point the installation to download chrome from our own mirror instead of the default.

kuisathaverat commented 4 years ago

None of these options works 

RUN npm set puppeteer_skip_download true \
  && npm set puppeteer_skip_chromium_download true \
  && npm config set puppeteer_skip_download true -g \
  && npm config set puppeteer_skip_chromium_download true -g \
  && PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true npm install

so I will implement a retry for the install