Open peanutfun opened 2 weeks ago
🙌 Thanks for analysis and elaboration!
I guess we'll apply the proposed prune_hazard
function to all wildfire datasets and upload the purged files then.
Are we sure it is a problem of wildfire alone or need we check whether other hazard objects are affected too?
Describe the bug We had noticed some strange patterns before when plotting the wildfire data from the API. However, the statistics on the
Hazard.intensity.data
looked fine, so we assumed a plotting issue. Turns out, theHazard.intensity
is not in canonical format, meaning that multiple entries inintensity.data
map to the same entry in theintensity
matrix. This causes erroneously high intensities for about 6% of all centroids.To Reproduce Steps to reproduce the behavior/error:
data
maximum.max
will bring it into canonical format.Output:
Background
The reason for this is that multiple entries in
intensity.data
point to the same matrix entries ofintensity
.csr_matrix
supports this, and sums up these values.Output:
The canonical format can be explicitly reached by calling
sum_duplicates()
. Summing the values is the default behavior and causes the exacerbated intensities.How to solve this
The following code prunes the doubled entries and calls an aggregation method. Unfortunately, the doubled entries are actually not doubled (as you can see above) but show different values. It is thus unclear how they should be merged into a single sensible value. It is possible to take the mean or the max.
I want to make clear that this is not just a fix. The following function modifies the hazard data. There seems to have been some underlying problem with merging the original wildfire data or with the hazard definition that should be fixed in order to avoid this inconsistent hazard definition from the start.
Result:![240613-wf-issue-02](https://github.com/CLIMADA-project/climada_python/assets/34276446/26070368-1a91-413b-b705-588efae67229)
Climada Version: develop
System Information (please complete the following information):
@emanuel-schmid @chahank @samluethi