Error in metrics_summary when class has no predicted values / variance for khat

[cycloxslug]

The metrics_summary function fails "error in off_diag * matrix : non-conformable arrays" when there are classes that are observed but not predicted. I would think the sensible default would be to include a row of zeros? Is there a way to workaround this?

Also, am I missing the value for variance of khat? It does not seem to be reported with khat.

[adriancorrendo]

Thanks, @cycloxslug !

None of the metrics would handle missing values. I left that in purpose. You'll need all the "predicted:observed" pairs complete in order to compute a metric value.

And you're right, metric doesn't offer an estimate for variance of khat. Are you using this for remote sensing? For the variance of khat you'll need to run a stratified random sampling or similar, which would be more complex than the original scope of estimating a variety of metrics. Metrica was designed to be used by users from a wide variety of backgrounds (and this stratification could be too specific).

Nonetheless, you may obtain a fair estimate of uncertainty (for any metric) by following a resampling and iteration of the metric function over it (e.g. bootstrapping).

I believe a code like this may help. It will produce a df with one estimate of khat per boot resample. Then you can extract a summary of the distribution (e.g., median, mean, sd, quantiles, etc.):

# Boostrapping for khat
 boot_khat <- 
 # n = no. of resamples
     function(data, pred, obs, n = 1000, ...) {
    # Allow customized column names
       p <- rlang::enquo(pred)
      o <- rlang::enquo(obs)

boot_id <- NULL 
boots <- NULL

output <- data %>%  
  dplyr::select(!!y, !!x, ...) %>%
  # Create a data frame with the boot resamples (1 per row)
  tidyr::expand_grid(boot_id = seq(1, n, by = 1)) %>%
  dplyr::group_by(boot_id, ...) %>%
  tidyr::nest(boots = c(!!o, !!p)) %>% 
  dplyr::mutate(boots = boots %>% 
                  purrr::map(function(boots) 
                    dplyr::slice_sample(boots, 
                                        replace = TRUE, n = nrow(boots)))) %>% 
  # Create iteration column
  dplyr::mutate(
    iter = map(boots, purrr::possibly(
      .f = ~ metrica::khat(data = ., pred = !!p, obs = !!o)),
      otherwise = NULL, quiet = TRUE)) %>%
  dplyr::select(-boots) %>% 
  tidyr::unnest(cols = iter) %>% 
  dplyr::ungroup()

return(output)

}

[cycloxslug]

Thanks for the feedback and approach - I am using this for remote sensing (sidescan sonar classification). I actually figured out how to force the classes to include empty classes - code all entries as factors, then all factors are included in the contingency table/confusion matrix.

As for khat variance - is bootstrapping necessary? Congalton and Green 2019 suggest the Delta method as an approximation. in Chapter 3 (Congalton, R. G., and K. Green. 2019. Assessing the Accuracy of Remotely Sensed Data: Principles and Practices, Third Edition, 3rd edition. CRC Press.)

Other implementations of Kappa calculate variance/confidence intervals and statistical significance via z-score (see vcd::Kappa() ).

[adriancorrendo]

Cool.

The metrica functions cannot handle missing values, you may just need to pass an na.omit or similar to run the functions. Metrica does not use matrices as inputs, just tabular format.

For the variance, the bootstrapping will be necessary as the metrics functions do not handle groups. That's something you can easily do outside the function. The variance you mention is estimated via resampling, so bootstrapping would be one of the most robust alternatives. Or you can also estimate one khat value per group you have and derive quantities from that.

Best, Adrian