Closed randytpierce closed 5 months ago
We encountered an alert on Feb 22 and investigation turned up... I did look into the log message for this error and indeed it is due to a query timeout while querying for a region (metadata). The log file is in this archive (you can find the appropriate archive in /data-ingest/data/xfer/archive by comparing the dates of the tar.gz files) ... /data-ingest/data/xfer/archive/success-job_v01_metar_partial__sums_surface_model_hrrrrap130_1708611006.tar.gz
And this is the error...
2024-02-22T14:25:32+0000 [ERROR] vxdata
._default.METAR\n WHERE type='MD'\n and docType='region'\n and subset='COMMON'\n and version='V01'\n and name='E_US'", 'parameters': '{"client_context_id":"e273cf-d29c-ab48-fc8f-41c30db8743250","metrics":false,"readonly":true,"statement":"SELECT geo.bottom_right.lat as br_lat,\n geo.bottom_right.lon as br_lon,\n geo.top_left.lat as tl_lat,\n geo.top_left.lon as tl_lon\n FROM vxdata
._default.METAR\n WHERE type=\'MD\'\n and docType=\'region\'\n and subset=\'COMMON\'\n and version=\'V01\'\n and name=\'E_US\'","timeout":"119500ms"}', 'context_type': 'QueryErrorContext'}), C Source=/couchbase-python-client/src/n1ql.cxx:277>)
From which I extracted the queries...
SELECT geo.bottom_right.lat as br_lat,
geo.bottom_right.lon as br_lon,
geo.top_left.lat as tl_lat,
geo.top_left.lon as tl_lon
FROM vxdata
._default.METAR
WHERE type = 'MD'
and docType = 'region'
and subset = 'COMMON'
and version = 'V01'
and name = 'E_US'
AND
SELECT geo.bottom_right.lat as br_lat,
geo.bottom_right.lon as br_lon,
geo.top_left.lat as tl_lat,
geo.top_left.lon as tl_lon
FROM vxdata
._default.METAR
WHERE type = 'MD'
AND docType='region'
AND subset='COMMON'
AND version='V01'
which are valid queries and work otherwise.
So we are seeing query timeouts against adb-cb1 from the python client in the ingest code.
There was some retry code added to a couple of problematic areas but this problem seems to have been resolved. We are not getting these alerts anymore.
We have gotten a couple of prometheus alerts indicating record count differences. Looking into one and comparing the timestamp to the archives it leads to this archive file ...
and opening that archive file with vim I look at this log ...
and I find these errors ....
These indicate a timeout problem talking to the database on adb-cb1. This also happened with connections and we fixed that problem with a retry. Perhaps the same thing is needed here, although it would be good to know why we are getting timeouts talking to the database.