Is "calcPredictionAccuracy" working correctly?

[kasper2619]

Hi.

Thank you for a great package.

I cannot get the output of "calcPredictionAccuracy" to make sense. Traning and evaluation of the recommender models works well and produces fine results.

However, when I use calcPredictionAccuracy I get only zero values in the TP columns. That does not align with the results I see when I train the model.

Can you shed some light over what I might be doing wrong, if anything? Reproducible example is provided below. See how the TP columns is consistently 0 even though the recommender itself should be performing well:

   TP FP FN TN N precision recall TPR       FPR
1   0  2  3  1 6         0      0   0 0.66
2   0  2  3  1 6         0      0   0 0.66
3   0  2  3  1 6         0      0   0 0.66
4   0  2  3  1 6         0      0   0 0.66
5   0  3  2  1 6         0      0   0 0.75
6   0  2  3  1 6         0      0   0 0.66
7   0  3  2  1 6         0      0   0 0.75
8   0  2  3  1 6         0      0   0 0.66
9   0  2  3  1 6         0      0   0 0.66
10  0  2  3  1 6         0      0   0 0.66

# load recommenderlab
library(recommenderlab)

# create a vector with binary entries
vec1 <- c(1,1,1,1,0,1,0,1,1,1)
vec2 <- c(0,0,0,0,1,0,1,0,0,0)

# write to dataframe
dat <- data.frame(
  p1 = vec1,
  p2 = vec2,
  p3 = vec1,
  p4 = vec2,
  p5 = vec1
) 

# convert to matrix
dat <- as.matrix(
  dat
)

# create binary rating matrix
dat_train <- as(dat, "binaryRatingMatrix")

# define algorithms to test
algorithms <- list(
  "Jaccard K1" = list(name="UBCF", param=list(nn=1, method = "Jaccard", verbose = T)),
  "Jaccard K2" = list(name="UBCF", param=list(nn=10, method = "Jaccard", verbose = T)),
  "Jaccard K3" = list(name="UBCF", param=list(nn=100, method = "Jaccard", verbose = T)),
  "Most Popular" = list(name="POPULAR", param=NULL),
  "Random" = list(name="RANDOM")
)

# set up evaluations scheme
es <- recommenderlab::evaluationScheme(
  data = dat_train,
  method = "cross-validation",
  k = 5,
  train = 0.7,
  given = -1,
)

# evaluate
results <- recommenderlab::evaluate(
  es,
  algorithms,
  type = "topNList",
  n=seq(1,3,1)
)

# assess resutls
res <- avg(results)
res <- results[["Jaccard K2"]]

# assess results
plot(results, "prec/rec", annotate=2, legend="topleft")
plot(results, annotate=c(1,3), legend="bottomright")

# we choose jackard K1 settings
rec <- Recommender(
  dat_train, 
  method = "UBCF",
  parameter = list(
    nn=10
  )
)
getModel(rec)

# predict
pre <- predict(rec, dat_train, n = 3)
pre

# as list
as(pre, "list")

# se prediction acuracy on user level
acc <- calcPredictionAccuracy(
  pre,
  data = dat_train,
  byUser = T,
  given = -1
)

acc

[kasper2619]

@mhahsler: Did you have time to consider the above question? :)

No rush but I would really like understand if I am doing anything wrong here...

[mhahsler]

Hi,

sorry for the delay. The way you have this set up results in:

> as(dat_train, "list")[1:3]
$`1`
[1] "p1" "p3" "p5"

$`2`
[1] "p1" "p3" "p5"

$`3`
[1] "p1" "p3" "p5"

> as(pre, "list")[1:3]
$`1`
[1] "p2" "p4"

$`2`
[1] "p2" "p4"

$`3`
[1] "p2" "p4"

There are no true positives and that is why TP is zero. You should train on training data and test against test data using an evaluation scheme. The relevant part from ? calcPredictionAccuracy:

data(MSWeb)
MSWeb10 <- sample(MSWeb[rowCounts(MSWeb) >10,], 50)

e <- evaluationScheme(MSWeb10, method="split", train=0.9,
    k=1, given=3)
e

## create a user-based CF recommender using training data
r <- Recommender(getData(e, "train"), "UBCF")

## create predictions for the test data using known ratings (see given above)
p <- predict(r, getData(e, "known"), type="topNList", n=10)
p

calcPredictionAccuracy(p, getData(e, "unknown"), given=3)
calcPredictionAccuracy(p, getData(e, "unknown"), given=3, byUser = TRUE)

Please give it a try and let me know if the results make more sense. I need to test if the all-but-x schemes work correctly.

[mhahsler]

I think I have found the issue in the code relating to calcPredictionAccuracy with negative values for given. I have added a bug fix to the development version on GitHub.

[kasper2619]

Cool.

I will test it out :)

[kasper2619]

Works like a charm now.

Thx :)