Stage2 CPU processing falling behind due to too much output (2D/3D layer writing)

retoomey / RAPIO

Real Time Algorithm Parameter and IO

GNU Lesser General Public License v3.0

2 stars 0 forks source link

Stage2 CPU processing falling behind due to too much output (2D/3D layer writing) #12

Open karencooper7 opened 4 months ago

karencooper7 commented 4 months ago

Need to speed up fusion stage2 --- can't write 2D and 3D both out or we fall behind.

Tree? Threading?

retoomey commented 4 months ago

Since we're more CPU dependent than w2merger the time to write affects us. So one idea was to use a thread pool and spawn N writers, but Netcdf/hdf5 seems to not be thread safe due to internal globals. Two other ideas include: 1. Using N dedicated writer processes using a shared memory queue. 2. Writing only 3D in stage2 and letting something else create the 2D. Two seems easiest so may explore that option.

retoomey commented 3 months ago

Implemented the 'tile' partitioning method (w2merger method) here: 43b5eef and here: 97c7d41. Since we use a global grid, we can still do the 'tree' method but that will require 3D algs to run on a single CONUS 3D cube.