problem with input precision in check_tbl_value_col_sum1()

[vpnagraj]

My team and I are currently preparing forecasts for the 2023-24 FluSight season. Ahead of the first submission, we have tested the hubValidations::validate_submissions() function. Most of the validations behave as expected. However, we have encountered some unexpected behavior with the "sum to 1" check for the "pmf" output type.

Our test submission file includes two cases where the values appear to sum to 1, but because of handling of floating point numbers (https://stackoverflow.com/questions/9508518/why-are-these-numbers-not-equal) the logic in check_tbl_value_col_sum1() (https://github.com/Infectious-Disease-Modeling-Hubs/hubValidations/blob/main/R/check_tbl_value_col_sum1.R#L52) flags these as not equal.

I've included a reprex below.

We have tried rounding down to 3 digits (as shown in the reprex) and still see the flag raised for some locations/horizons. Short of rounding down to 2 digits, I'm not sure if there is anything we can do on our end to resolve this problem. Would you all consider updating the logic in check_tbl_value_col_sum1() to use all.equal() or dplyr::near() or another approach? If not do you have any suggestions for us to try? I would expect that if we are having this issue, other submitters might run into it as well.

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

## works
tribble(~value,
        "0.752",
        "0.183",
        "0.065") %>%
  mutate(value = as.numeric(value)) %>%
  mutate(not_sum1 = !sum(.data[["value"]]) == 1L)
#> # A tibble: 3 × 2
#>   value not_sum1
#>   <dbl> <lgl>   
#> 1 0.752 FALSE   
#> 2 0.183 FALSE   
#> 3 0.065 FALSE

## does not work
tribble(~value,
        "0.818",
        "0.180",
        "0.002") %>%
  mutate(value = as.numeric(value)) %>%
  mutate(not_sum1 = !sum(.data[["value"]]) == 1L)
#> # A tibble: 3 × 2
#>   value not_sum1
#>   <dbl> <lgl>   
#> 1 0.818 TRUE    
#> 2 0.18  TRUE    
#> 3 0.002 TRUE

## looking more closely
tst <-
  tribble(~value,
          "0.818",
          "0.180",
          "0.002") %>%
  mutate(value = as.numeric(value))

## what are the values
tst$value
#> [1] 0.818 0.180 0.002
## what is the sum of the values
sum(tst$value)
#> [1] 1
## is the sum of the values == 1L
sum(tst$value) == 1L
#> [1] FALSE
## how about the same test using all.equal
all.equal(sum(tst$value),1L)
#> [1] TRUE
## and with dplyr::near
dplyr::near(sum(tst$value),1L)
#> [1] TRUE

## now with all.equal in the pipeline
tribble(~value,
        "0.818",
        "0.180",
        "0.002") %>%
  mutate(value = as.numeric(value)) %>%
  mutate(not_sum1 = !all.equal(sum(.data[["value"]]),1L))
#> # A tibble: 3 × 2
#>   value not_sum1
#>   <dbl> <lgl>   
#> 1 0.818 FALSE   
#> 2 0.18  FALSE   
#> 3 0.002 FALSE

^{Created on 2023-10-06 with reprex v2.0.2}

[elray1]

Hi -- thanks for filing this issue! I agree that we should account for numerical/floating point precision issues in this check. Please expect an update early next week.

[annakrystalli]

Thank you so much for your detailed bug report @vpnagraj ! This was indeed an oversight on our part and has been fixed in PR #53 (packade version v0.0.0.9003).

I've included an additional test for the probabilities that were causing errors for you. If you could try validating your submission gain using the latest package version and confirm it is fixed for you that would be greatly appreciated!

Interestingly, the behaviour you described is somewhat system specific as the original approach also worked for me on my machine with your example. But this is a such a well known issue that the more robust approach across systems is of course required.

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

tst <-
    tribble(~value,
            "0.818",
            "0.180",
            "0.002") %>%
    mutate(value = as.numeric(value))

sum(tst$value) == 1L
#> [1] TRUE

all.equal(sum(tst$value), 1L)
#> [1] TRUE

^{Created on 2023-10-09 with reprex v2.0.2}

Session info

``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.2.1 (2022-06-23) #> os macOS Ventura 13.5.2 #> system aarch64, darwin20 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Europe/Athens #> date 2023-10-09 #> pandoc 3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> cli 3.6.1 2023-03-23 [1] CRAN (R 4.2.0) #> digest 0.6.33 2023-07-07 [1] CRAN (R 4.2.0) #> dplyr * 1.1.3 2023-09-03 [1] CRAN (R 4.2.0) #> evaluate 0.22 2023-09-29 [1] CRAN (R 4.2.0) #> fansi 1.0.4 2023-01-22 [1] CRAN (R 4.2.0) #> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.2.0) #> fs 1.6.3 2023-07-20 [1] CRAN (R 4.2.0) #> generics 0.1.3 2022-07-05 [1] CRAN (R 4.2.1) #> glue 1.6.2 2022-02-24 [1] CRAN (R 4.2.0) #> htmltools 0.5.6 2023-08-10 [1] CRAN (R 4.2.0) #> knitr 1.42 2023-01-25 [1] CRAN (R 4.2.0) #> lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.2.0) #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.0) #> pillar 1.9.0 2023-03-22 [1] CRAN (R 4.2.0) #> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.2.0) #> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.2.0) #> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.2.0) #> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.2.0) #> R.oo 1.25.0 2022-06-12 [1] CRAN (R 4.2.0) #> R.utils 2.12.2 2022-11-11 [1] CRAN (R 4.2.0) #> R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.0) #> reprex 2.0.2 2022-08-17 [3] CRAN (R 4.2.0) #> rlang 1.1.1 2023-04-28 [1] CRAN (R 4.2.0) #> rmarkdown 2.21 2023-03-26 [1] CRAN (R 4.2.0) #> rstudioapi 0.14 2022-08-22 [1] CRAN (R 4.2.1) #> sessioninfo 1.2.2 2021-12-06 [3] CRAN (R 4.2.0) #> styler 1.7.0 2022-03-13 [1] CRAN (R 4.2.0) #> tibble 3.2.1 2023-03-20 [1] CRAN (R 4.2.0) #> tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.2.0) #> utf8 1.2.3 2023-01-31 [1] CRAN (R 4.2.0) #> vctrs 0.6.3 2023-06-14 [1] CRAN (R 4.2.0) #> withr 2.5.0 2022-03-03 [1] CRAN (R 4.2.0) #> xfun 0.39 2023-04-20 [1] CRAN (R 4.2.0) #> yaml 2.3.7 2023-01-23 [1] CRAN (R 4.2.0) #> #> [1] /Users/Anna/Library/R/arm64/4.2/library #> [2] /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/site-library #> [3] /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

Thank you again for reporting! As we are still in testing phase of the framework, we would greatly appreciate any and all bug reports as well as any suggestions on how functionality can be improved to meet team's needs.

[vpnagraj]

Thank you all for such a quick turn around on this fix!

Confirming that when I test the validation with v0.0.0.9003 I am now seeing a successful check (i.e., no issues with any of the pmf forecasts summing to 1 due to precision).