Before looping though sites and imts we pre-calculate the weights, branch_hash_table, and component_branches
weights numpy array
branch_hash_table List[List[str]] (1_000_000[4])
component_branches: List[HazardComponentBranch]
We want to distribute calc_aggregation() across processes with MultiProcess. We should share the common objects to avoid copying the arrays and lists (may not need to bother with component_branches as it can be calculated on the fly quite quickly)
Note that for the full NSHM_v1.0.4 the weight array and hash table are quite small and perhaps this is not necessary:
chrisdc@glacier:~/.../DEV/toshi-hazard-post$ NZSHM22_THS_LOCAL_DIR=/data/toshi_hazard_store/pq-CDC2/ poetry run thp aggregate demo/hazard_v2.toml
warning openquake module dependency not available, maybe you want to install
with nzshm-model[openquake]
Toshi Hazard Post: hazard curve aggregation
=================================================
2024-04-26 15:33:15,329 - toshi_hazard_post.version2.aggregation - INFO - getting sites . . .
2024-04-26 15:33:15,329 - toshi_hazard_post.version2.aggregation - INFO - getting logic trees . . .
2024-04-26 15:33:15,406 - toshi_hazard_post.version2.aggregation - INFO - calculating weights and branch hash table . . .
2024-04-26 15:33:15,406 - toshi_hazard_post.version2.logic_tree - DEBUG - generating composite branches
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - time to build weight array and hash table 28.31 seconds
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - Size of weight array: 7MB
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - Size of hash table: 8MB
Before looping though sites and imts we pre-calculate the
weights
,branch_hash_table
, andcomponent_branches
weights
numpy arraybranch_hash_table
List[List[str]] (1_000_000[4])component_branches
: List[HazardComponentBranch]We want to distribute
calc_aggregation()
across processes withMultiProcess
. We should share the common objects to avoid copying the arrays and lists (may not need to bother withcomponent_branches
as it can be calculated on the fly quite quickly)Note that for the full
NSHM_v1.0.4
the weight array and hash table are quite small and perhaps this is not necessary: