search

NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a developer, I need to find and refactor the Netcdf writer classes to open files as infrequently as possible #201

Open epag opened 3 weeks ago

epag commented 3 weeks ago

Author Name: Chris (Chris) Original Redmine Issue: 61079, https://vlab.noaa.gov/redmine/issues/61079 Original Date: 2019-03-07

Constantly opening and closing Netcdf files when writing data scales horrendously. An example of this is in @NetcdfOutputWriter.getOrigin@ where the output file is opened for every metric value that needs to be written. If just about every conceivable USGS location for short range data for all metrics are written, that will result in close to 3 million writes from that function alone.

Any other instances of extraneous opening/closing of netcdf files need to be found and refactored as well.

@wres.grid.io.griddedReader.GriddedReader@ uses a @NetcdfDataset@ to act as a sort of object pool for gridded files. It might also be worth using that for the output writer as well.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-07T23:33:26Z

Probably still an issue.