search

GNS-Science / toshi-hazard-post

Hazard post-processing pipeline as serverless AWS infrastructure.
MIT License
0 stars 1 forks source link

Share common logic tree data across processes #31

Open chrisdicaprio opened 4 months ago

chrisdicaprio commented 4 months ago

Before looping though sites and imts we pre-calculate the weights, branch_hash_table, and component_branches

We want to distribute calc_aggregation() across processes with MultiProcess. We should share the common objects to avoid copying the arrays and lists (may not need to bother with component_branches as it can be calculated on the fly quite quickly)

Note that for the full NSHM_v1.0.4 the weight array and hash table are quite small and perhaps this is not necessary:

chrisdc@glacier:~/.../DEV/toshi-hazard-post$ NZSHM22_THS_LOCAL_DIR=/data/toshi_hazard_store/pq-CDC2/ poetry run thp aggregate  demo/hazard_v2.toml 
warning openquake module dependency not available, maybe you want to install
                with nzshm-model[openquake]
Toshi Hazard Post: hazard curve aggregation
=================================================
2024-04-26 15:33:15,329 - toshi_hazard_post.version2.aggregation - INFO - getting sites . . .
2024-04-26 15:33:15,329 - toshi_hazard_post.version2.aggregation - INFO - getting logic trees . . . 
2024-04-26 15:33:15,406 - toshi_hazard_post.version2.aggregation - INFO - calculating weights and branch hash table . . . 
2024-04-26 15:33:15,406 - toshi_hazard_post.version2.logic_tree - DEBUG - generating composite branches
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - time to build weight array and hash table 28.31 seconds
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - Size of weight array: 7MB
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - Size of hash table: 8MB