Feature/roc

[FariborzDaneshvar-NOAA]

Added ROC_single_run.py to calculate stats (i.e., hit, miss, ... and POD,FAR), and plot ROC (based on POD and FAR).

Command with example args:

python ROC_single_run.py \
--storm_name="Florence" \
--storm_year=2018 \
--leadtime=48 \
--prob_nc_path='/nhc/nhc_runs_v1/florence_2018_OFCL_48hr_korobov_30/setup/ensemble.dir/analyze/linear_k1_p1_n0.025/probabilities.nc' \
--obs_df_path='/home/Fariborz.Daneshvar/notebooks/NextGen_P_Surge/PSurge_obs_xGEOID20b.csv'

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA is the CSV file aggregate of the USGS stations?

Also about about the location of the new file, I agree that it's a "post" script. But in this case it needs to be evaluated in the "prep" environment, like the analyze_ensemble.py script. The post refers to the env created for deterministic run (before probabilistic run was implemented). Maybe we can just put this under prep/files as well and at some point think about reorganizing everything.

What do you think?

[FariborzDaneshvar-NOAA]

@

@FariborzDaneshvar-NOAA is the CSV file aggregate of the USGS stations?

This is the aggregate of all observation points I got from the NHC team (including COOPS and USGS) for retrospective storms. It has two attributes for two vdatums (NAVD88, and xGEOID20b). Maybe putting it in the static_data ?!

[FariborzDaneshvar-NOAA]

Also about about the location of the new file, I agree that it's a "post" script. But in this case it needs to be evaluated in the "prep" environment, like the analyze_ensemble.py script. The post refers to the env created for deterministic run (before probabilistic run was implemented). Maybe we can just put this under prep/files as well and at some point think about reorganizing everything.

What do you think?

100% agree! I put it here just for the reference. Do you want me to move it to singularity/prep/files?

I was also thinking of integrating it into analyze_ensemble, but first wanted to add it to EnsemblePerturbation, and call it from there. Any thoughts?

[SorooshMani-NOAA]

Do you want me to move it to singularity/prep/files

Yes please.

Any thoughts?

It's a good idea. I agree it's a good idea to integrate. So are you thinking about adding it as a separate example in ensembleperturbation like what we have for klpc?

[FariborzDaneshvar-NOAA]

It's a good idea. I agree it's a good idea to integrate. So are you thinking about adding it as a separate example in ensembleperturbation like what we have for klpc?

No, I was thinking about adding these two functions:

• make_ROC_curve_inputs() function in ensembleperturbation/uncertainty_quantification/surrogate.py. Inputs will be 1) probabilities.nc that will be created earlier with probability_field_from_surrogate(), and 2) obs_csv (from static_data for now).
• plot_ROC_curve() function in ensembleperturbation/plotting/surrogate.py with one input (netcdf file created in the previous step)

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA I moved it to /prep/files/ as you suggested. You might want to close this PR until I make the next change! thanks for sharing your thoughts.

[SorooshMani-NOAA]

I missed your earlier comment about CSV ... we can put that in static data, but if we want to put anything related to that in ensemble perturbation we should generalize it so that data can be fetched using your stormevents usgs module

I like your suggestion for adding the functions in the ensemble perturbation, let's just make sure we deal with the USGS aggregate input.

[SorooshMani-NOAA]

Let's keep it like this, we'll merge it when you're ready

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA I forgot about this ... is it OK to merge this?

[FariborzDaneshvar-NOAA]

is it OK to merge this?

Yes please, thanks.

[SorooshMani-NOAA]

From my side I see the same file added to both prep and post directories. I accept the merge, but then I'll cleanup in #51 where I'm reorganizing stuff

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA Thanks! Sorry I forgot to remove the duplicate one from post!