search

unity-sds / unity-project-management

Container repo for project management (projects, epics, etc)
Apache License 2.0
0 stars 1 forks source link

Define infrastructure requirements for SITC use case #163

Closed mike-gangl closed 1 month ago

mike-gangl commented 2 months ago

View Determine SITC algorithm(s) for forward processing for information on algorithms.

Algorithm information and resources required can be found here.

Algorithm Inputs (Number) Outputs (Number) Resources
NASA VIIRS L1 NASA 2h L0 VIIRS (1),NASA 2h L0 Spacecraft (1) 6-minute VIIRS GEO/L1B (20) 1 CPU | 4 GB RAM / Core (TBD)
CLDMSK_L2_VIIRS 6-minute VIIRS GEO/L1B (1)GDAS-0Zf (2) NISE (1)OISST (1) 6-minute CLDMSK_L2_VIIRS (1) 1 CPU | 4 GB RAM / Core (TBD)
CLDMSK_G3 6-minute CLDMSK_L2_VIIRS (1) 6-minute CLDMSK_G3 (1) 1 CPU | 4 GB RAM / Core
CLDMSK_D3 6-minute CLDMSK_G3 (240) CLDMSK_D3 (1) 1 CPU | 4 GB RAM / Core
CLDMSK_M3 CLDMSK_D3 (28|30|31) CLDMSK_M3 (1) 1 CPU | 4 GB RAM / Core

For forward stream processing- we require 10-20 cores at all times. That comes out to probably 2-3 machines. We might need to determine the network/disk IO capacity with the DAAC if the processes become IO bound.

I'd look at 3 m7i.2xlarge to start (24 cores). Another alternative are the m6in.2xlarge if there is ALOT of network usage.

I'm avoiding the C instances for now since they don't scale the memory enough (2GB/core) and the memory-optimized instances (R) scale it too much at 8GB/core.