search

OCHA-DAP / pa-anticipatory-action

Code and documentation for analytical work on OCHA Anticipatory Action pilots.
GNU General Public License v3.0
14 stars 0 forks source link

NER trigger report #314

Closed joseepoirier closed 1 year ago

joseepoirier commented 1 year ago

Hi both, I would love to get a second set of eyes on this before it gets shipped. The Rmd itself is mostly transferrable to other countries at this point although some manual adjustments are likely necessary for trigger mechanisms that don't have the exact same structure as NER (number of triggers and activation timepoints.)

@caldwellst Hopefully the code itself is now cleaned up enough to be a quick PR to do but if you are short on time, could you please just take a look at the text/contents instead of the code, as Pauline does not yet have much experience with AA triggers?

Thanks!

joseepoirier commented 1 year ago

Should have mentioned: only analyses/ner/NER_trigger_report.Rmd and plotCIs.R need your attention. Thanks!

caldwellst commented 1 year ago

Hey Josée! Sorry, as you yourself mentioned, no time at the moment to review the code but have had a look at the document output. Really beautiful plotting and graphics, nice work!

The thing that stood out to me while reviewing was the performance of the observational trigger. I was wondering, is the observational trigger 3 metrics only assessed based on years that the trigger 2 was not met (mirroring the way the framework is set up)? If so, I think visualising somehow the performance of the "Package 2" triggering together would be very valuable. As is, it is strange to see that the observational trigger performs much worse than the 2 predictive triggers, and imagine that would be the main question many people might have when looking at the document.

Wondering as well, do we want to list what the thresholds are for the triggers? It seems there's a lot of other details about the individual triggers but that is missing. As well, how was performance of the trigger 3 assessed? Just against CHIRPS I assume? Might be good to specify that for clarity.

joseepoirier commented 1 year ago

Thanks @caldwellst !

Wondering as well, do we want to list what the thresholds are for the triggers? It seems there's a lot of other details about the individual triggers but that is missing.

Good catch! I myself only noticed after I sent it out. The ones that were missing are now displayed in the activation timepoints table.

As well, how was performance of the trigger 3 assessed? Just against CHIRPS I assume? Might be good to specify that for clarity.

Right, that would be good to explain. It's validated against historical bad years.

Visualising somehow the performance of the "Package 2" triggering together would be very valuable.

Good point. It may not apply to all frameworks but we should have a way to display that. It's akin to the need to present the performance of T1 and T2 across months. Will explore options.

joseepoirier commented 1 year ago

@PaulineNimo I will transfer and adapt the code so the trigger_report_template reflects the most recent version. I'll also work with @turnerm to make sure the code to generate the confidence intervals is well aligned with the markdown to generate the reports (column names, what is generated through analysis vs. in a markdown chunk, etc.)

Merging this PR as Leonardo has OKed the final report.

PaulineNimo commented 1 year ago

@PaulineNimo I will transfer and adapt the code so the trigger_report_template reflects the most recent version. I'll also work with @turnerm to make sure the code to generate the confidence intervals is well aligned with the markdown to generate the reports (column names, what is generated through analysis vs. in a markdown chunk, etc.)

Merging this PR as Leonardo has OKed the final report.

Thanks. I haven't been able to go through this yet.