search

OCHA-DAP / pa-anticipatory-action

Code and documentation for analytical work on OCHA Anticipatory Action pilots.
GNU General Public License v3.0
14 stars 0 forks source link

Tcd metrics #321

Closed caldwellst closed 1 year ago

caldwellst commented 1 year ago

Implemented the Chad trigger card based on the discussions we've had regarding the rule of three and bootstrapping.

caldwellst commented 1 year ago

Leonardo's comments and then my responses, from Slack:

Thanks! I am surprised that the activation for trigger 1 and 2 is quite low. Have you taken into account the fact that we have 2 trigger points instead of just one? We have seen that this has a significant impact as the forecasts for different months are not necessarily highly correlated

Yes, this accounts for the multiple triggers. Individually they have about 5% chance of reaching that threshold.

Shouldn’t the target period always be Jul-Sep for trigger 1 and 2?

Yes, typo! Will address.

Can you add somewhere a description (maybe under Key messages - Shock detection) of how the actual shocks are estimated?

This is added beneath the all shocks graph under notes. This is where we listed in the Niger report.

What is the expected cost in the table? Is it the expected annual cost on average? If that’s the case let’s make it explicit

The expected cost is again following the Niger template, where it’s the central likelihood of activation multiplied by the cost. How would you recommend making it explicit? (Leonardo said yes let's make it explicit)

caldwellst commented 1 year ago

I added just an asterisk next to Expected Cost to indicate it's covered in the notes, but I'm sure there's a better solution.

caldwellst commented 1 year ago

Have updated to call it "Expected Annual Cost" baed on feedback from Leonardo.

joseepoirier commented 1 year ago

Sorry Seth I don't see an email notifying me of this PR. Will review shortly.

Two things I already noticed: (1) there is no learning table, which there should be even if you don't report the full set of metrics that we had for NER (2) plots: i believe I found a way to keep the plots on Gdrive Data. I'll test it on the template and let you know if/when we can move them.

caldwellst commented 1 year ago

What do you mean by learning table? The Learning Opportunities table is still there in the report at the bottom, but let me know if it's something else I missed. I took the Niger report and directly changed that so hopefully nothing missing.

joseepoirier commented 1 year ago

What do you mean by learning table? The Learning Opportunities table is still there in the report at the bottom, but let me know if it's something else I missed. I took the Niger report and directly changed that so hopefully nothing missing.

Ha no, it's there indeed. I had forgotten it was stored on Data. I'll take a closer look at this PR asap. Sorry it's confusing this is my third of fourth version of a report in 2 days! 😂

caldwellst commented 1 year ago

Thanks Josée! Have gone through and addressed. Not sure what you mean about me thinking we did something wrong? Would you mind pointing me to this? I thought might be in the Learning Opportunities section but couldn't figure out exactly what that might be.

I like the thought about how to report on observational triggers that are frequently observed. Actually not the case here, we only observe once like in Niger, but interesting stuff about Malawi. I made an issue about it so we don't lose the thought when this PR is closed!

joseepoirier commented 1 year ago

Hey Seth,

A doc/line reference would have been helpful, sorry. In tcd_iri_seas_forecast_probability.md, now line 172. You write "I am therefore wondering if we are doing this correctly.."

It would be good for you, @turnerm and I to brainstorm performance metrics for continuous triggers. Monica's already done a lot of thinking on this for sudden-onset triggers and her input would be valuable as we figure out the case of observational triggers.

Lastly, is there an explanation for why the forecasts were global and not "cropped" to Chad? If not, it seems relevant to include.

caldwellst commented 1 year ago

Aaah I see. That's very old code. All of that was actually Tinka originally, I was just making a few amendments to the file to get it running properly on my machine. And things like that comment are also from her. And agreed on the discussion and brainstorming, I think will be most useful to do that when we have a specific case to work on, like Malawi. Have clarified why using global forecasts for the likelihood of activation!