Flaky Mac test log upload

rickeylev commented 1 month ago

Over the last few days I've noticed that the upload step after builds finish has been rather flaky. This is happening pretty often. Usuallly a retry fixes it, but sometimes it take 4-5 retries.

https://buildkite.com/bazel/rules-python-python/builds/7863#018fd0ee-0089-4499-ac64-44a7e1a0c507

A build will be successful, but then have an error like below:

Traceback (most recent call last):
  File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 4461, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 4429, in main
    execute_commands(
  File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 1437, in execute_commands
    _ = future.result()
        ^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 2646, in upload_test_logs_from_bep
    execute_command(
  File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 2709, in execute_command
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/tmprzybvb6m/bazelci-agent', 'artifact', 'upload', '--debug', '--delay=5', '--mode=buildkite', '--build_event_json_file=/tmp/tmprzybvb6m/test_bep.json']' returned non-zero exit status 1.

meteorcloudy commented 1 month ago

/cc @coeuvre

meteorcloudy commented 2 weeks ago

Probably obsolete due to https://github.com/bazelbuild/continuous-integration/issues/1981

bazelbuild / continuous-integration

Flaky Mac test log upload #1963