V8 CI on LinuxONE (s390x) fails to fetch dependencies

[richardlau]

Starting 2 August, node-test-commit-v8-linux has started to fail for rhel8-s390x.

e.g. https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=rhel8-s390x,v8test=v8test/6105/console

10:39:14 ________ running 'python3 third_party/depot_tools/update_depot_tools_toggle.py --disable' in './v8'
10:39:15 ________ running 'python3 build/landmines.py --landmine-scripts tools/get_landmines.py' in './v8'
11:21:16 ________ running 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth --bucket chromium-clang-format -s buildtools/linux64/clang-format.sha1' in './v8'
11:21:16 Failed to fetch file gs://chromium-clang-format/b42097ca924d1f1736a5a7806068fed9d7345eb4 for buildtools/linux64/clang-format. [Err: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
11:21:16   warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
11:21:16 INFO 0802 05:49:13.469579 retry_util.py] Retrying request, attempt #1...
11:21:16 INFO 0802 05:50:15.649107 retry_util.py] Retrying request, attempt #2...
11:21:16 INFO 0802 05:51:20.085787 retry_util.py] Retrying request, attempt #3...
11:21:16 INFO 0802 05:52:30.012090 retry_util.py] Retrying request, attempt #4...
11:21:16 INFO 0802 05:53:42.362467 retry_util.py] Retrying request, attempt #5...
11:21:16 INFO 0802 05:55:09.449953 retry_util.py] Retrying request, attempt #6...
11:21:16 INFO 0802 05:56:41.551597 retry_util.py] Retrying request, attempt #7...
11:21:16 INFO 0802 05:58:13.661195 retry_util.py] Retrying request, attempt #8...
11:21:16 INFO 0802 05:59:45.769585 retry_util.py] Retrying request, attempt #9...
11:21:16 INFO 0802 06:01:17.873443 retry_util.py] Retrying request, attempt #10...
11:21:16 INFO 0802 06:02:49.970290 retry_util.py] Retrying request, attempt #11...
11:21:16 INFO 0802 06:04:22.019599 retry_util.py] Retrying request, attempt #12...
11:21:16 INFO 0802 06:05:54.119333 retry_util.py] Retrying request, attempt #13...
11:21:16 INFO 0802 06:07:26.219733 retry_util.py] Retrying request, attempt #14...
11:21:16 INFO 0802 06:08:58.341854 retry_util.py] Retrying request, attempt #15...
11:21:16 INFO 0802 06:10:30.458922 retry_util.py] Retrying request, attempt #16...
11:21:16 INFO 0802 06:12:02.569424 retry_util.py] Retrying request, attempt #17...
11:21:16 INFO 0802 06:13:34.679120 retry_util.py] Retrying request, attempt #18...
11:21:16 INFO 0802 06:15:06.779235 retry_util.py] Retrying request, attempt #19...
11:21:16 INFO 0802 06:16:38.889257 retry_util.py] Retrying request, attempt #20...
11:21:16 INFO 0802 06:18:10.989735 retry_util.py] Retrying request, attempt #21...
11:21:16 INFO 0802 06:19:43.100408 retry_util.py] Retrying request, attempt #22...
11:21:16 OSError: None.
11:21:16 ]
11:21:16 0> Downloading buildtools/linux64/clang-format@b42097ca924d1f1736a5a7806068fed9d7345eb4...
11:21:16 0> Failed to fetch file gs://chromium-clang-format/b42097ca924d1f1736a5a7806068fed9d7345eb4 for buildtools/linux64/clang-format, skipping. [Err: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
11:21:16   warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
11:21:16 INFO 0802 05:49:13.469579 retry_util.py] Retrying request, attempt #1...
11:21:16 INFO 0802 05:50:15.649107 retry_util.py] Retrying request, attempt #2...
11:21:16 INFO 0802 05:51:20.085787 retry_util.py] Retrying request, attempt #3...
11:21:16 INFO 0802 05:52:30.012090 retry_util.py] Retrying request, attempt #4...
11:21:16 INFO 0802 05:53:42.362467 retry_util.py] Retrying request, attempt #5...
11:21:16 INFO 0802 05:55:09.449953 retry_util.py] Retrying request, attempt #6...
11:21:16 INFO 0802 05:56:41.551597 retry_util.py] Retrying request, attempt #7...
11:21:16 INFO 0802 05:58:13.661195 retry_util.py] Retrying request, attempt #8...
11:21:16 INFO 0802 05:59:45.769585 retry_util.py] Retrying request, attempt #9...
11:21:16 INFO 0802 06:01:17.873443 retry_util.py] Retrying request, attempt #10...
11:21:16 INFO 0802 06:02:49.970290 retry_util.py] Retrying request, attempt #11...
11:21:16 INFO 0802 06:04:22.019599 retry_util.py] Retrying request, attempt #12...
11:21:16 INFO 0802 06:05:54.119333 retry_util.py] Retrying request, attempt #13...
11:21:16 INFO 0802 06:07:26.219733 retry_util.py] Retrying request, attempt #14...
11:21:16 INFO 0802 06:08:58.341854 retry_util.py] Retrying request, attempt #15...
11:21:16 INFO 0802 06:10:30.458922 retry_util.py] Retrying request, attempt #16...
11:21:16 INFO 0802 06:12:02.569424 retry_util.py] Retrying request, attempt #17...
11:21:16 INFO 0802 06:13:34.679120 retry_util.py] Retrying request, attempt #18...
11:21:16 INFO 0802 06:15:06.779235 retry_util.py] Retrying request, attempt #19...
11:21:16 INFO 0802 06:16:38.889257 retry_util.py] Retrying request, attempt #20...
11:21:16 INFO 0802 06:18:10.989735 retry_util.py] Retrying request, attempt #21...
11:21:16 INFO 0802 06:19:43.100408 retry_util.py] Retrying request, attempt #22...
11:21:16 OSError: None.
11:21:16 ]
11:21:16 Downloading 1 files took 2520.994796 second(s)
11:21:16 Error: Command 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth --bucket chromium-clang-format -s buildtools/linux64/clang-format.sha1' returned non-zero exit status 1 in ./v8
11:21:16 Failed to fetch file gs://chromium-clang-format/b42097ca924d1f1736a5a7806068fed9d7345eb4 for buildtools/linux64/clang-format. [Err: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
11:21:16   warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
11:21:16 INFO 0802 05:49:13.469579 retry_util.py] Retrying request, attempt #1...
11:21:16 INFO 0802 05:50:15.649107 retry_util.py] Retrying request, attempt #2...
11:21:16 INFO 0802 05:51:20.085787 retry_util.py] Retrying request, attempt #3...
11:21:16 INFO 0802 05:52:30.012090 retry_util.py] Retrying request, attempt #4...
11:21:16 INFO 0802 05:53:42.362467 retry_util.py] Retrying request, attempt #5...
11:21:16 INFO 0802 05:55:09.449953 retry_util.py] Retrying request, attempt #6...
11:21:16 INFO 0802 05:56:41.551597 retry_util.py] Retrying request, attempt #7...
11:21:16 INFO 0802 05:58:13.661195 retry_util.py] Retrying request, attempt #8...
11:21:16 INFO 0802 05:59:45.769585 retry_util.py] Retrying request, attempt #9...
11:21:16 INFO 0802 06:01:17.873443 retry_util.py] Retrying request, attempt #10...
11:21:16 INFO 0802 06:02:49.970290 retry_util.py] Retrying request, attempt #11...
11:21:16 INFO 0802 06:04:22.019599 retry_util.py] Retrying request, attempt #12...
11:21:16 INFO 0802 06:05:54.119333 retry_util.py] Retrying request, attempt #13...
11:21:16 INFO 0802 06:07:26.219733 retry_util.py] Retrying request, attempt #14...
11:21:16 INFO 0802 06:08:58.341854 retry_util.py] Retrying request, attempt #15...
11:21:16 INFO 0802 06:10:30.458922 retry_util.py] Retrying request, attempt #16...
11:21:16 INFO 0802 06:12:02.569424 retry_util.py] Retrying request, attempt #17...
11:21:16 INFO 0802 06:13:34.679120 retry_util.py] Retrying request, attempt #18...
11:21:16 INFO 0802 06:15:06.779235 retry_util.py] Retrying request, attempt #19...
11:21:16 INFO 0802 06:16:38.889257 retry_util.py] Retrying request, attempt #20...
11:21:16 INFO 0802 06:18:10.989735 retry_util.py] Retrying request, attempt #21...
11:21:16 INFO 0802 06:19:43.100408 retry_util.py] Retrying request, attempt #22...
11:21:16 OSError: None.
11:21:16 ]
11:21:16 0> Downloading buildtools/linux64/clang-format@b42097ca924d1f1736a5a7806068fed9d7345eb4...
11:21:16 0> Failed to fetch file gs://chromium-clang-format/b42097ca924d1f1736a5a7806068fed9d7345eb4 for buildtools/linux64/clang-format, skipping. [Err: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
11:21:16   warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
11:21:16 INFO 0802 05:49:13.469579 retry_util.py] Retrying request, attempt #1...
11:21:16 INFO 0802 05:50:15.649107 retry_util.py] Retrying request, attempt #2...
11:21:16 INFO 0802 05:51:20.085787 retry_util.py] Retrying request, attempt #3...
11:21:16 INFO 0802 05:52:30.012090 retry_util.py] Retrying request, attempt #4...
11:21:16 INFO 0802 05:53:42.362467 retry_util.py] Retrying request, attempt #5...
11:21:16 INFO 0802 05:55:09.449953 retry_util.py] Retrying request, attempt #6...
11:21:16 INFO 0802 05:56:41.551597 retry_util.py] Retrying request, attempt #7...
11:21:16 INFO 0802 05:58:13.661195 retry_util.py] Retrying request, attempt #8...
11:21:16 INFO 0802 05:59:45.769585 retry_util.py] Retrying request, attempt #9...
11:21:16 INFO 0802 06:01:17.873443 retry_util.py] Retrying request, attempt #10...
11:21:16 INFO 0802 06:02:49.970290 retry_util.py] Retrying request, attempt #11...
11:21:16 INFO 0802 06:04:22.019599 retry_util.py] Retrying request, attempt #12...
11:21:16 INFO 0802 06:05:54.119333 retry_util.py] Retrying request, attempt #13...
11:21:16 INFO 0802 06:07:26.219733 retry_util.py] Retrying request, attempt #14...
11:21:16 INFO 0802 06:08:58.341854 retry_util.py] Retrying request, attempt #15...
11:21:16 INFO 0802 06:10:30.458922 retry_util.py] Retrying request, attempt #16...
11:21:16 INFO 0802 06:12:02.569424 retry_util.py] Retrying request, attempt #17...
11:21:16 INFO 0802 06:13:34.679120 retry_util.py] Retrying request, attempt #18...
11:21:16 INFO 0802 06:15:06.779235 retry_util.py] Retrying request, attempt #19...
11:21:16 INFO 0802 06:16:38.889257 retry_util.py] Retrying request, attempt #20...
11:21:16 INFO 0802 06:18:10.989735 retry_util.py] Retrying request, attempt #21...
11:21:16 INFO 0802 06:19:43.100408 retry_util.py] Retrying request, attempt #22...
11:21:16 OSError: None.
11:21:16 ]
11:21:16 Downloading 1 files took 2520.994796 second(s)
11:21:16 
11:21:16 
11:21:16 
11:21:16 Warnings:
11:21:16 Conflicting directory ./v8/third_party/fp16/src moved to ./_bad_scm/v8/third_party/fp16/src5sq06epk.
11:21:16 Conflicting directory ./v8/third_party/abseil-cpp moved to ./_bad_scm/v8/third_party/abseil-cppnqidpmxw.
11:21:16 Conflicting directory ./v8/third_party/jinja2 moved to ./_bad_scm/v8/third_party/jinja2p9k44c33.
11:21:16 Conflicting directory ./v8/third_party/googletest/src moved to ./_bad_scm/v8/third_party/googletest/src48ws40yo.
11:21:16 Conflicting directory ./v8/third_party/markupsafe moved to ./_bad_scm/v8/third_party/markupsafexttd4006.
11:21:16 Conflicting directory ./v8/third_party/zlib moved to ./_bad_scm/v8/third_party/zlibmvbdgj43.
11:21:16 Hook 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth --bucket chromium-clang-format -s buildtools/linux64/clang-format.sha1' took 2521.07 secs
11:21:16 Checking out depot_tools.
11:21:16 Using depot tools in /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools
11:21:16 Initializing temporary git repository in v8.
11:21:16 Fetching dependencies.
11:21:16 Uninitializing temporary git repository
11:21:16 >> Cleaning up /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/.git
11:21:16 Traceback (most recent call last):
11:21:16   File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/../../tools/v8/fetch_deps.py", line 100, in <module>
11:21:16     FetchDeps(sys.argv[1])
11:21:16   File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/../../tools/v8/fetch_deps.py", line 82, in FetchDeps
11:21:16     subprocess.check_call([sys.executable, gclient, "sync", "--spec", spec],
11:21:16   File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
11:21:16     raise CalledProcessError(retcode, cmd)
11:21:16 subprocess.CalledProcessError: Command '['/usr/bin/python3', '/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/gclient.py', 'sync', '--spec', "solutions = [{'name': 'v8', 'url': 'https://chromium.googlesource.com/v8/v8.git', 'deps_file': 'DEPS', 'managed': False, 'custom_deps': {'v8/base/trace_event/common': None, 'v8/test/benchmarks/data': None, 'v8/testing/gmock': None, 'v8/test/mozilla/data': None, 'v8/test/test262/data': None, 'v8/test/test262/harness': None, 'v8/third_party/android_ndk': None, 'v8/third_party/android_sdk': None, 'v8/third_party/catapult': None, 'v8/third_party/colorama/src': None, 'v8/third_party/fuchsia-sdk': None, 'v8/third_party/instrumented_libraries': None, 'v8/tools/luci-go': None, 'v8/tools/swarming_client': None, 'v8/third_party/qemu-linux-x64': None}}]"]' returned non-zero exit status 2.
11:21:16 make: *** [Makefile:309: v8] Error 1

This only appears to be happening on rhel8-s390x -- the x64 and ppc64le builds are passing.

[richardlau]

11:21:16 Failed to fetch file gs://chromium-clang-format/b42097ca924d1f1736a5a7806068fed9d7345eb4 for buildtools/linux64/clang-format. [Err: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()

I'm slightly surprised to be downloading something with linux64 in its path on s390x, but I don't think this is new -- I think this is coming from https://github.com/nodejs/node/blob/90dea9e3e003945f113aaca09c45f963fe566a47/deps/v8/DEPS#L498-L509 and that isn't recent.

[richardlau]

As a quick test I tried disabling hooks in fetch_deps.py:

diff --git a/tools/v8/fetch_deps.py b/tools/v8/fetch_deps.py
index b7e628608a..51f9cc8ef1 100755
--- a/tools/v8/fetch_deps.py
+++ b/tools/v8/fetch_deps.py
@@ -79,7 +79,7 @@ def FetchDeps(v8_path):
     env["PATH"] = depot_tools + os.pathsep + env["PATH"]
     gclient = os.path.join(depot_tools, "gclient.py")
     spec = "solutions = %s" % GCLIENT_SOLUTION
-    subprocess.check_call([sys.executable, gclient, "sync", "--spec", spec],
+    subprocess.check_call([sys.executable, gclient, "sync", "--spec", spec, "--nohooks"],
                            cwd=os.path.join(v8_path, os.path.pardir),
                            env=env)
   except:

https://ci.nodejs.org/job/node-test-commit-v8-linux/6118/

Unfortunately while that does appear to have allowed s390x to fetch dependencies and start building, it has broken the x64 build.

I think it might be possible to set custom_hooks in the spec we pass to gclient sync, but I haven't yet been able to get that working on s390x.

[richardlau]

11:21:16 Failed to fetch file gs://chromium-clang-format/b42097ca924d1f1736a5a7806068fed9d7345eb4 for buildtools/linux64/clang-format. [Err: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()

I'm slightly surprised to be downloading something with linux64 in its path on s390x, but I don't think this is new -- I think this is coming from https://github.com/nodejs/node/blob/90dea9e3e003945f113aaca09c45f963fe566a47/deps/v8/DEPS#L498-L509 and that isn't recent.

Although in v8-canary this appears to have been moved to linux64-format in buildtools: https://chromium.googlesource.com/chromium/src/buildtools.git/+/19268125543e152a215808913dacafe77ecae63f/DEPS#58

[richardlau]

hmm and now I'm not seeing the problem again on the CI 😕(🎉 ?). Will pick this up again on Monday.

[richardlau]

All the weekend runs passed. Going to optimistically close.