search

voltrondata-labs / arrow-benchmarks-ci

Benchmarks CI for Apache Arrow project
MIT License
0 stars 5 forks source link

Schedule-and-publish job keeps getting canceled #164

Closed austin3dickey closed 1 year ago

austin3dickey commented 1 year ago

Similar to #148. We keep retrying a certain compare runs endpoint which never returns inside the DB query timeout:

[2023-09-26 14:05:02,434] INFO in http: unexpected response. code: 500 (retryable), body bytes: <unexpected exception, please report this:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.QueryCanceled: canceli ...>

Since the 2023-09-26 02:50:00Z run.

austin3dickey commented 1 year ago

Merged a temporary fix (https://github.com/conbench/conbench/pull/1492) but can't deploy it until https://github.com/conbench/conbench/pull/1493 is fixed.

austin3dickey commented 1 year ago

I deployed the fix and let the pipeline run for 45 minutes to clear out the backlog. It seems to be working now!