Closed douglatornell closed 3 weeks ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 77.64%. Comparing base (
1eca15e
) to head (6cda311
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Successfully tested in production on arbutus
since 1-Jun-2024. No worker stalls, but we sometimes went 30 days without a stall with the previous code. Merging because this PR doesn't break anything, and there is reason to hope that it improves the reliability of the workers.
These changes were motivated by the intermittent stalls of the
make_ww3_current_file
andmake_ww3_wind_file
workers that have plagued operations for the past year. The changes reflect the many things that have been learned about usingxarray.opern_dataset()
,xarray.open_mfdataset()
, anddask
since the workers were written.h5netcdf
package for dataset reads in the hop of improving the reliability of themake_ww3_current_file
andmake_ww3_wind_file
workersdask
processing and explicitly call the.compute()
before writing the datasets to force read processing withh5netcdf
in the process-baseddask
schedulernetcdf4
as the engine for dataset writing. This avoids incompatibilities in the resulting file that arise if it is written usingh5netcdf
main()
function docstrings re: issue #121