Transient IndexingIntegrationTest.test_indexing failure for IT job in a recent develop build

https://gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/-/jobs/94204
======================================================================
ERROR: test_indexing (integration_test.IndexingIntegrationTest.test_indexing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_job_helpers.py", line 91, in do_query
    query_job._begin(retry=retry, timeout=timeout)
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/job/query.py", line 1371, in _begin
    super(QueryJob, self)._begin(client=client, retry=retry, timeout=timeout)
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/job/base.py", line 693, in _begin
    api_response = client._call_api(
                   ^^^^^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 816, in _call_api
    return call()
           ^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 204, in retry_target
    return target()
           ^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.Conflict: 409 POST https://bigquery.googleapis.com/bigquery/v2/projects/platform-hca-dev/jobs?prettyPrint=false: Already Exists: Job platform-hca-dev:us-central1.a5a69895-cd95-4d18-82e1-72dbbcaeec48

Location: None
Job ID: a5a69895-cd95-4d18-82e1-72dbbcaeec48

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/ucsc/azul/test/azul_test_case.py", line 254, in wrapped
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 504, in test_indexing
    notifications, fqids = self._prepare_notifications(catalog)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 1282, in _prepare_notifications
    list(starmap(update, self._list_bundles(catalog,
  File "/builds/ucsc/azul/test/integration_test.py", line 375, in _list_bundles
    source, prefix, new_fqids = self._list_partition_bundles(catalog, source)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 305, in _list_partition_bundles
    fqids = self.azul_client.list_bundles(catalog, source, partition_prefix)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/azulclient.py", line 228, in list_bundles
    return plugin.list_bundles(source, prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/plugins/repository/tdr.py", line 204, in list_bundles
    bundle_fqids = self._list_bundles(source, prefix)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/plugins/repository/tdr_hca/__init__.py", line 327, in _list_bundles
    current_bundles = self._query_latest_version(source.spec, f'''
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/plugins/repository/tdr_hca/__init__.py", line 344, in _query_latest_version
    iter_rows = self._run_sql(query)
                ^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/plugins/repository/tdr.py", line 225, in _run_sql
    return self.tdr.run_sql(query)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/terra.py", line 493, in run_sql
    job: QueryJob = bigquery.query(query)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 3411, in query
    return _job_helpers.query_jobs_insert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_job_helpers.py", line 114, in query_jobs_insert
    future = do_query()
             ^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_job_helpers.py", line 100, in do_query
    query_job = client.get_job(
                ^^^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 2096, in get_job
    resource = self._call_api(
               ^^^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 816, in _call_api
    return call()
           ^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 204, in retry_target
    return target()
           ^^^^^^^^
  File "/build/.venv/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.NotFound: 404 GET https://bigquery.googleapis.com/bigquery/v2/projects/platform-hca-dev/jobs/a5a69895-cd95-4d18-82e1-72dbbcaeec48?projection=full&prettyPrint=false: Not found: Job platform-hca-dev:a5a69895-cd95-4d18-82e1-72dbbcaeec48

----------------------------------------------------------------------
Ran 17 tests in 134.408s

FAILED (errors=1, skipped=1)
make: *** [Makefile:247: integration_test] Error 1
section_end:1718070104:step_script
[0Ksection_start:1718070104:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1718070105:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 1
DataBiosphere / azul

Transient IndexingIntegrationTest.test_indexing failure for IT job in a recent develop build #6328
