At the moment we're only working in Minnesota, and we are pulling from the netCDF by cell and by time period (opening each of the 6 netCDFs one time each), adding the burn-in and burn out and then writing to .csv. Upon model execution, the correct .csv is copied into each simulation directory.
When we scale up to the full footprint, we should switch to an approach where we instead pull out the data by tile and by time period, storing it in feather files. That way we'd generate (n_tiles3) files, instead of (n_cells3) files. That approach will mean that within the execution code, instead of copying the correct file for each simulation to the simulation directory, we'll need to need to read in the feather file for the correct tile and time period, filter it to the correct cell, and then write the filtered data to the simulation directory. The DS team does not anticipate any issues with multiple targets branches simultaneously accessing the same feather file.
Note that the nitty gritty of our approach may be influenced by #41
At the moment we're only working in Minnesota, and we are pulling from the netCDF by cell and by time period (opening each of the 6 netCDFs one time each), adding the burn-in and burn out and then writing to .csv. Upon model execution, the correct .csv is copied into each simulation directory.
When we scale up to the full footprint, we should switch to an approach where we instead pull out the data by tile and by time period, storing it in feather files. That way we'd generate (n_tiles3) files, instead of (n_cells3) files. That approach will mean that within the execution code, instead of copying the correct file for each simulation to the simulation directory, we'll need to need to read in the feather file for the correct tile and time period, filter it to the correct cell, and then write the filtered data to the simulation directory. The DS team does not anticipate any issues with multiple
targetsbranches simultaneously accessing the same feather file.Note that the nitty gritty of our approach may be influenced by #41