469 update retrofit dataset

IN-CORE / pyincore

pyIncore is a component of IN-CORE. It is a python package consisting of two primary components: 1) a set of service classes to interact with the IN-CORE web services, and 2) IN-CORE analyses . The pyIncore allows users to apply various hazards to infrastructure in selected areas, propagating the effect of physical infrastructure damage and loss of functionality to social and economic impacts.

Mozilla Public License 2.0

25 stars 7 forks source link

469 update retrofit dataset #470

Closed longshuicy closed 9 months ago

longshuicy commented 11 months ago

add additional function to update the existing inventory dataset with additional retrofit and mapping entry key information. https://github.com/IN-CORE/pyincore/pull/470/files#diff-d1b1cb98d647ccb4ebd9f558b2894472d298769fdd3c2bde8d9d4e00205e4de0R73

Since we use fiona object and looping through fiona is too slow, I used geopandas dataframe to merge and udpate; but then save to shapefile so we can resume using fiona to carry out normal fragility matching and calculation

To test:

Use this notebook to test SLC retrofit strategy SLC_retrofit_test.zip
Use this notebook to tests Galveston Galveston_retrofit_test.zip
RUN tests/pyincore/analyses/buildingdamage/test_buildingdamage_retrofit.py and see results in the retrofit folder

Note:

Joplin takes 31 seconds;
Lumberton takes 61 seconds.
SLC if only retrofit 5000+ building with inner join total runtime: 158 seconds;
Galveston retrofitting 1500 buildings combined damage takes 115 seconds

longshuicy commented 9 months ago

Made one update to use inner join when joining with retrofit strategies and mapping; this will make sure only the building that has retrofit strategy will get computed again. Hence help with the performance.

ylyangtw commented 9 months ago

Encountered FutureWarning in Galveston notebook:

/pyincore/utils/dataprocessutil.py:281: FutureWarning: The behavior of DataFrame.idxmax with all-NA values, or any-NA and skipna=False, is deprecated. In a future version this will raise ValueError
  max_key = dmg_result[dmg_states].idxmax(axis=1)

Just committed a fix if that's ok. Feel free to revert it if it affects anything. Other than this, both notebooks and test work.

Some test records from my end: Joplin Tornado Retrofit execution time: 11.40898 seconds Lumberton Flood Retrofit execution time: 44.84685 seconds Galveston notebook runtime: 71 & 20 SLC notebook: 162 & 52

Good job on the hardwork!

longshuicy commented 9 months ago

Encountered FutureWarning in Galveston notebook:
/pyincore/utils/dataprocessutil.py:281: FutureWarning: The behavior of DataFrame.idxmax with all-NA values, or any-NA and skipna=False, is deprecated. In a future version this will raise ValueError
  max_key = dmg_result[dmg_states].idxmax(axis=1)
Just committed a fix if that's ok. Feel free to revert it if it affects anything. Other than this, both notebooks and test work.

Some test records from my end: Joplin Tornado Retrofit execution time: 11.40898 seconds Lumberton Flood Retrofit execution time: 44.84685 seconds Galveston notebook runtime: 71 & 20 SLC notebook: 162 & 52

Good job on the hardwork!

Thanks for the fix and thorough test!