Estimate computational demands for short- and long-term tasks

[sigmafelix]

The ultimate objective of this project is to make scalable GIS computation easier to adequately versed R/GIS users (e.g., master's students in epidemiology or geography). However, the short-term goal is to serve the {SET}group's NRT-AP project to compute required covariates nationwide (i.e., mainland United States). This issue is made to estimate the scale of covariate computation for the NRT-AP project as well as computational scales that potential users would need to work with.

NRT-AP

• Mainland US: ~8 million sq km; split the mainland into standard rectangular regions

  • 1 km spatial resolution: 8 million points
  • 500 m spatial resolution: 32 million points
  • 2 km spatial resolution: 2 million points
  • Splitting the mainland into rectangular regions would be a viable option as the target points will be generated regularly. Rectangular shapes have advantages of the reduction in subsets of base data for covariates (e.g., raster data)

• Checklist for scaling up the computation

  • Memory footprint
  • Multicore or multisession parallelization
  • Storage read speed and memory bandwidth + inter-node network bandwidth
  • Total data amount to distribute to each node
  • Configuration of preprocessing and main processing assets

Brainstorming possible use cases

• Irregular individual residential locations in highly dense urban areas
• Temporally regular sample points along trajectories
• Sample points with spatially varying density

Function ideas

• [ ] Takes inputs of hardware configuration, data dimensions and complexity (e.g., shape complexity in vector data), data size and then determine the optimal splits to reduce running time
• [x] Merge adjacent computational grids with the small number of target points (grid_merge; implemented)

[sigmafelix]

09/14/2023 update

• The spatial resolutions listed on the covariate table have the range from 30 meters (NLCD) to 0.65 by 0.50 arc degrees (approx. 72 kilometers by 55 kilometers in the equator)
• For model fitting, site locations are the first target to compute covariates (N=1,087). Splitting sites by rectangular grids may not result in the preferable performance compared to the split-with-grids computation at regularly gridded points.
• Regional splits that approximate rectangular grids would be reasonable.
  • Modified census regions: based on Census Regional Divisions

Region	States
Northwest	WA-OR-MT-ID-WY
Southwest Pacific	CA-NV
Southwest Mountain	AZ-NM-CO-UT
West North Central North	ND-SD-MN
West North Central South	NE-KS-IA-MO
East North Central	WI-IL-IN-MI-OH
West South Central West	OK-TX
West South Central East	AR-LA-MS
East South Central	KY-TN-AL
South Atlantic South	SC-GA-FL
South Atlantic North	NC-VA-WV-DC-MD-DE
Middle Atlantic	NY-PA-NJ
New England	RI-CT-MA-VT-NH-ME

[sigmafelix]

Brainstorming--

Factors

• Data resolution: vector and raster
• Memory + CPU demand per run at each feature $\propto$ F(total vertices in a vector dataset, total raster cells involved)
• future strategy: multicore, multisession, or future.callr? How to handle global objects?
• Algorithmic properties (i.e., exactextractr::exact_extract): preprocessing rasters to stack to enhance performance (which consumes a lot of memory; but extraction does not consume as much memory as preprocessing)
• "Sweet spot" for attaining the optimal performance
• Startup/ignition overhead when future::multisession or future.batchtools::batchtools_slurm is used

Helper function: est_demand() or consult()

• Reading computation code then suggest (or even auto-switch) the "better" way
• Question: how to standardize demands for custom workflows?