NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

remove dependency on primary index by converting queries that are agiqanst the id field to gets. #335

Closed randytpierce closed 6 months ago

randytpierce commented 6 months ago

remove dependency on primary index by converting queries that are agiqanst the id field to gets. These are queries for job documents which aren't even all that large. By changing them from queries to data fetches they are a lot more efficient and don't require either a special index or a primary index.

github-actions[bot] commented 6 months ago

Code Coverage

Package Line Rate Branch Rate Health
vxingest 33% 38%
vxingest.builder_common 27% 15%
vxingest.ctc_to_cb 12% 1%
vxingest.grib2_to_cb 13% 1%
vxingest.netcdf_to_cb 13% 1%
vxingest.partial_sums_to_cb 12% 1%
vxingest.utilities 30% 32%
Summary 16% (490 / 3033) 6% (40 / 712)
randytpierce commented 6 months ago

This is really strait-forward and should be covered by the integration tests. NOTE I could have also changed ...

Select id as ingest_document_ids
from `vxdata`._default.METAR
where meta().id = "MD:V01:METAR:obs:ingest:netcdf"

to

Select id as ingest_document_ids
from `vxdata`._default.METAR
USE KEYS ["JOB:V01:METAR:CTC:CEILING:MODEL:OPS"]

but I think by-passing the query processor is probably more efficient.