search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.25k stars 3.46k forks source link

[CI][JS] Javascript Release verification tests are extremely slow #38155

Open pitrou opened 11 months ago

pitrou commented 11 months ago

Describe the bug, including details regarding any error messages, version, and platform.

In the release verification builds, the Javascript tests are extremely slow. For example, in the build below, the tests took an entire hour. https://github.com/ursacomputing/crossbow/actions/runs/6447710112/job/17504244473#step:6:737

$ cross-env NODE_NO_WARNINGS=1 gulp test
[12:55:48] Using gulpfile /arrow/js/gulpfile.js
[12:55:48] Starting 'test'...
[12:55:48] Starting 'test:ts'...
[13:01:11] Finished 'test:ts' after 5.38 min
[13:01:11] Starting 'test:src'...
[13:07:11] Finished 'test:src' after 5.98 min
[13:07:11] Starting 'test:apache-arrow'...
[13:12:16] Finished 'test:apache-arrow' after 5.08 min
[13:12:16] Starting 'test:es5:cjs'...
[13:17:17] Finished 'test:es5:cjs' after 5 min
[13:17:17] Starting 'test:es2015:cjs'...
[13:22:10] Finished 'test:es2015:cjs' after 4.87 min
[13:22:10] Starting 'test:esnext:cjs'...
[13:26:46] Finished 'test:esnext:cjs' after 4.6 min
[13:26:46] Starting 'test:es5:esm'...
[13:32:12] Finished 'test:es5:esm' after 5.43 min
[13:32:12] Starting 'test:es2015:esm'...
[13:37:29] Finished 'test:es2015:esm' after 5.27 min
[13:37:29] Starting 'test:esnext:esm'...
[13:42:37] Finished 'test:esnext:esm' after 5.13 min
[13:42:37] Starting 'test:es5:umd'...
[13:46:40] Finished 'test:es5:umd' after 4.03 min
[13:46:40] Starting 'test:es2015:umd'...
[13:50:29] Finished 'test:es2015:umd' after 3.82 min
[13:50:29] Starting 'test:esnext:umd'...
[13:54:16] Finished 'test:esnext:umd' after 3.77 min
[13:54:16] Finished 'test' after 58 min
Done in 3509.95s.

Component(s)

Continuous Integration, JavaScript, Release

pitrou commented 11 months ago

@domoritz @trxcllnt

pitrou commented 11 months ago

CI resources are shared between all builds of Arrow (and some of them are paid for by Voltron Data), so we should strive to be careful with build times.

Also cc @assignUser

assignUser commented 11 months ago

I am aware that these take a long time but I have no insights as I am rather unfamiliar with the underlying build system. It would be great if we can optimize this somehow