Closed kinto-b closed 3 years ago
To answer the issue title - no :P
This was a hacky rough attempt to incorporate the original frequency (to allow more leeway for high frequency responses, since for e.g. a jump from 2 to 12 percent is usually very different in checking than a jump from 72 to 82).
Should probably use an actual statistical measure instead.
Yeah, good point.
What about chisq.test()
?
Coming back to this, chisq.test()
is very sensitive to differences. For instance:
df1 <- data.frame(
a = sample(1:5, 100000, TRUE),
b = sample(c(rep(1:5, 5), 1:3), 100000, TRUE),
c = sample(c(rep(1:5, 25), 1:3), 100000, TRUE),
d = sample(c(rep(1:5, 125), 1:3), 100000, TRUE)
)
df1 %>% group_by(level = a) %>% summarise(n_a = n()) %>%
left_join(
df1 %>% group_by(level = b) %>% summarise(n_b = n()), "level"
) %>%
left_join(
df1 %>% group_by(level = c) %>% summarise(n_c = n()), "level"
) %>%
left_join(
df1 %>% group_by(level = d) %>% summarise(n_d = n()), "level"
)
#> # A tibble: 5 x 5
#> level n_a n_b n_c n_d
#> <int> <int> <int> <int> <int>
#> 1 1 20180 21234 20241 20026
#> 2 2 19932 21382 20398 20159
#> 3 3 19820 21494 20255 20050
#> 4 4 20031 17956 19654 19905
#> 5 5 20037 17934 19452 19860
chisq.test(table(df1$a), p = table(df1$b), rescale.p = TRUE)
#> Chi-squared test for given probabilities
#>
#> data: table(df1$a)
#> X-squared = 767.42, df = 4, p-value < 2.2e-16
chisq.test(table(df1$a), p = table(df1$c), rescale.p = TRUE)
#> Chi-squared test for given probabilities
#>
#> data: table(df1$a)
#> X-squared = 44.997, df = 4, p-value = 3.982e-09
chisq.test(table(df1$a), p = table(df1$d), rescale.p = TRUE)
#> Chi-squared test for given probabilities
#>
#> data: table(df1$a)
#> X-squared = 8.7539, df = 4, p-value = 0.06755
If we consider the p-values, we would never say that a
is similar to b
or c
. But to my eyes c
looks very similar to a
.
We could potentially use the chi-squared statistic directly. But a small chi-square value doesn't necessarily indicate that the distributions are similar, only that we can't confidently tell them apart: that could be because they are similar or because there aren't many data points.
Spoke to Andrew about this and he recommended using chi-square.
I'm now questioning this function. It might be best to leave similarity testing to users given that it's fairly hairy
@wilcoxa @tonoplast RE: similarity testing discussion this morning
@wilcoxa I'm leaning towards deprecating this one. I've slapped an experimental
badge on it anyway.
Once this is sorted out, 0.2.0 will be ready to review/merge/release. I think the other issues that are currently outstanding can wait
@kinto-b Yeah experimental
is good for the moment - worth keeping track of the progress made at the very least. There's definitely a need for this type of function, but needs some work to make this useful in the wild.
@gorcha I think this one should be deprecated before CRAN-ing
Yep, agreed
Suppose we compare the following,
How it currently works
What does
expect_similar()
do internally? First it does thishttps://github.com/socialresearchcentre/testdat/blob/9f3ef727756836853e3d8e2993428ff5848c3b9e/R/expect-datacomp.R#L33-L34
which yields
Then it does this
https://github.com/socialresearchcentre/testdat/blob/9f3ef727756836853e3d8e2993428ff5848c3b9e/R/expect-datacomp.R#L36-L40
which yields
How I thought it would work
But I was expecting it to run the comparison in this way:
which yields