search

NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

Seeing errors in METAR netcdf ingest processing. #313

Closed randytpierce closed 7 months ago

randytpierce commented 7 months ago

I am seeing errors like this... 

2024-01-29T18:26:49+0000 [INFO] <VxIngestManager-2> (vxingest.builder_common.ingest_manager): VxIngestManager-1: IngestManager - processing /public/data/madis/point/metar/netcdf/.prev
2024-01-29T18:26:49+0000 [INFO] <VxIngestManager-2> (vxingest.netcdf_to_cb.vx_ingest_manager): process_element - : start time: 1706552809
2024-01-29T18:26:49+0000 [ERROR] <VxIngestManager-2> (vxingest.netcdf_to_cb.netcdf_builder): NetcdfMetarObsBuilderV01: Exception with builder build_document: file_name: /public/data/madis/point/metar/netcdf/.prev
Traceback (most recent call last):
  File "/app/vxingest/netcdf_to_cb/netcdf_builder.py", line 344, in build_document
    self.ncdf_data_set = nc.Dataset(queue_element)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno 22] Invalid argument: '/public/data/madis/point/metar/netcdf/.prev'

in obs processing sometimes. Notice the "Exception with builder build_document: file_name: /public/data/madis/point/metar/netcdf/.prev" It seems to think it needs to process a file named ".prev"

randytpierce commented 7 months ago

This error happened because the python globbing that we were using in the file selection started returning entries that start with '.'. It seems that they didn't used to do that. I added additional filtering in the fet_file_list method to qualify the file names by their pattern. Their pattern is a date format string that defines how the filenames are mapped to timestamps. This is fixed in branch '313-seeing-errors-in-metar-netcdf-ingest-processing'