verifyBrier Error

[map579]

Hi! In using the verifyBrier function I received the following error:

Error in verifyBrier(anenX, obs, threshold, ensemble.func, ### baseline = ozoneForecasts) : all(ret.flt$check - ret.flt$bs < 1e-06) is not TRUE

The error occurred when using a forecast file that contained "NA" for the first 12 FLTs of data for the parameter of ozone concentrations, while parameters for temp, wind speed, and wind direction included data. When running the same script but with a slightly modified forecast file which contains zeroes for the first 12 FLTs of data for the parameter of ozone concentrations, instead of "NA", the function works as expected.

I can follow the script for verifyBrier, and in running the individual lines of verifyBrier, the verification::brier command completes in line 136. However, it seems as if the ret.flt$check values are all NA (for all monitors and all days).

I can email the input files and scripts that I am using, upon request. Thanks!

Mark

[Weiming-Hu]

Hi Mark,

Could you save your input to the function verfiyBrier as an RData and attach it here? I'm looking for a reproducible case so that I can help you debug.

If your input is too big, please try to subset them. And make sure the subset dataset still gives you the error.

Thanks

[map579]

Here you go, Weiming. Let me know if I can provide any other information about the data. Thank you! anenOutput2020-04-02_Run_18.zip

Mark

[Weiming-Hu]

Hi Mark,

Thank you for the dataset. It is very helpful.

Your dataset actually led me to dig a little bit deeper into verfication::brier and found out that they only support scalar climatological baseline value. In other words, the baseline is supposed to be a scalar. This is, in your case, not applicable because you are trying to compare two operational forecasts.

I'm afraid that I have to remove baseline from RAnEnExtra::verifyBrier since comparing between two operational models is almost always the case but not supported by the underlying package verification.

I would suggest you calculate brier scores for AnEn and your operational forecast model separately and then manually compare them.

I have provided my version of scripts:

library(RAnEnExtra)

# Load the subset data
load('~/Desktop/anenOutput2020-04-02_Run_18.RData')

# # Verify analogs as you normally would
scores_anen <- verifyBrier(sub.anen, sub.obs, threshold, ensemble.func)

# # Add a dimension to the end of the model so that the dimensions are consistent with analogs
dim(sub.baseline) <- c(dim(sub.baseline), 1)

# Verify baseline model as an one-member forecast model
scores_baseline <- verifyBrier(sub.baseline, sub.obs, threshold, baseline = T)

# Comparison plot
library(magrittr)
library(ggplot2)
library(reshape2)

df_plot <- data.frame(FLT = rownames(scores_anen),
                      AnEn = scores_anen[, 'bs'],
                      Baseline = scores_baseline[, 'bs']) %>%
  melt(measure.vars = c('AnEn', 'Baseline'), id.vars = 'FLT')

ggplot(data = df_plot) + geom_col(mapping = aes(x = FLT, y = value, fill = variable), position = 'dodge')

Thanks

[Weiming-Hu]

By the way, please update RAnEnExtra.

[Weiming-Hu]

The subset figure looks suspicious. But since it is just a subset, I will have you to let me know whether things will look differently when a larger dataset is used.

[map579]

After updating RAnEnExtra, I reran the verifyBrierfunction just as you provided in your example script above. Unfortunately, I receive a new error. I tried the function with a full dataset and the previously provided subset and I receive the same error each time:

> scores_anen <- verifyBrier(sub.anen, sub.obs, threshold, ensemble.func)
Error: lazy-load database 'C:/Users/spark/Documents/R/win-library/3.6/RAnEnExtra/R/RAnEnExtra.rdb' is corrupt
In addition: Warning messages:
1: restarting interrupted promise evaluation 
2: internal error -3 in R_decompress1 

I'll note that the installation of the updated RAnEnExtrapackage was successful with no hiccups.

[Weiming-Hu]

This is usually caused by unhandled processes and R is not able to decompress the file. Could you restart your R session and try again?

[map579]

That indeed worked, for both the subset and full dataset. Thanks! - Mark