Benchmarks

[Hydrotoast]

An extremely simple benchmarking script against fastfm. Currently this package is slower, but the RMSE is better given the same parameters.

# benchmarks/run_benchmarks.sh
Running fastFM benchmark
2.31317913225

real    0m0.719s
user    0m0.583s
sys 0m0.136s

Running FM Julia benchmark
INFO: Running tests on ml-100k dataset...
  0.268140 seconds (2.59 M allocations: 112.726 MB, 13.40% gc time)
INFO: Train dim: (2623,90570)
  0.029074 seconds (266.15 k allocations: 12.549 MB, 6.29% gc time)
INFO: Test dim: (2072,9430)
INFO: Test ML-100K - Training model...
  0.000082 seconds (9 allocations: 266.703 KB)
INFO: Learning Factorization Machines with gradient descent...
  0.019411 seconds (181.14 k allocations: 13.820 MB)
  0.107642 seconds (245.14 k allocations: 18.224 MB)
INFO: [SGD - Epoch 1] Evaluation: 2.0230283321900657
  0.134071 seconds (426.53 k allocations: 32.746 MB, 1.44% gc time)
  0.025456 seconds (181.14 k allocations: 13.820 MB, 30.27% gc time)
  0.026902 seconds (181.24 k allocations: 15.207 MB, 28.64% gc time)
INFO: [SGD - Epoch 2] Evaluation: 2.2867888783095762
  0.048056 seconds (362.45 k allocations: 29.721 MB, 19.36% gc time)
  0.018268 seconds (181.14 k allocations: 13.820 MB, 4.57% gc time)
  0.018944 seconds (181.20 k allocations: 15.204 MB, 4.40% gc time)
INFO: [SGD - Epoch 3] Evaluation: 2.4177370417848794
  0.039796 seconds (362.42 k allocations: 29.718 MB, 2.10% gc time)
  0.013573 seconds (181.14 k allocations: 13.820 MB)
  0.014718 seconds (181.20 k allocations: 15.204 MB, 3.83% gc time)
INFO: [SGD - Epoch 4] Evaluation: 2.4363909281989273
  0.038188 seconds (362.42 k allocations: 29.718 MB, 3.23% gc time)
  0.018767 seconds (181.14 k allocations: 13.820 MB, 3.77% gc time)
  0.026237 seconds (181.20 k allocations: 15.204 MB, 2.70% gc time)
INFO: [SGD - Epoch 5] Evaluation: 2.439853158936633
  0.053324 seconds (362.42 k allocations: 29.718 MB, 1.33% gc time)
  0.017076 seconds (181.14 k allocations: 13.820 MB)
  0.017690 seconds (181.20 k allocations: 15.204 MB)
INFO: [SGD - Epoch 6] Evaluation: 2.432807561391459
  0.041549 seconds (362.42 k allocations: 29.718 MB, 1.45% gc time)
  0.019991 seconds (181.14 k allocations: 13.820 MB, 3.49% gc time)
  0.020578 seconds (181.20 k allocations: 15.204 MB, 3.39% gc time)
INFO: [SGD - Epoch 7] Evaluation: 2.421628978406314
  0.041658 seconds (362.42 k allocations: 29.718 MB, 3.22% gc time)
  0.020645 seconds (181.14 k allocations: 13.820 MB, 2.86% gc time)
  0.021587 seconds (181.20 k allocations: 15.204 MB, 2.74% gc time)
INFO: [SGD - Epoch 8] Evaluation: 2.4748771072239863
  0.043771 seconds (362.42 k allocations: 29.718 MB, 1.35% gc time)
  0.018886 seconds (181.14 k allocations: 13.820 MB)
  0.019510 seconds (181.20 k allocations: 15.204 MB)
INFO: [SGD - Epoch 9] Evaluation: 2.479370380861987
  0.041199 seconds (362.42 k allocations: 29.718 MB, 1.42% gc time)
  0.017653 seconds (181.14 k allocations: 13.820 MB, 11.92% gc time)
  0.018239 seconds (181.20 k allocations: 15.204 MB, 11.53% gc time)
INFO: [SGD - Epoch 10] Evaluation: 2.48548420026441
  0.042015 seconds (362.42 k allocations: 29.718 MB, 7.78% gc time)
  0.807474 seconds (3.84 M allocations: 307.080 MB, 2.53% gc time)
INFO: Test ML-100K - Prediction over test dataset...
  0.001938 seconds (18.86 k allocations: 1.511 MB)
INFO: RMSE on test data: 1.5642163124864876

real    0m2.472s
user    0m2.334s
sys 0m0.143s

It seems that the majority of the time for this package is spent on the last iteration of SGD: 0.807474 seconds (3.84 M allocations: 307.080 MB, 2.53% gc time). Perhaps this should warrant some investigation.

[Hydrotoast]

References #6

[coveralls]

Coverage remained the same at 96.491% when pulling fc69d7d91ab69c2afd3e783fbffe5c3acf66d57d on Hydrotoast:benchmarks into 3016c1087bd0d46a901d954dd03942b9dcb9f7a9 on btwardow:master.

[btwardow]

Ok, I changed a little bit benchmark code and created PR to your branch for review :-)

[coveralls]

Coverage remained the same at 96.552% when pulling 80cb777d221c318500c2968a6186c1a8bb77f957 on Hydrotoast:benchmarks into b2ea74801503396e40f8ecbc1910d684f1f1133f on btwardow:master.

[coveralls]

Coverage remained the same at 96.552% when pulling 0653181b65fedd6fec2b672b76a1c9d5151eb1e9 on Hydrotoast:benchmarks into b2ea74801503396e40f8ecbc1910d684f1f1133f on btwardow:master.

[coveralls]

Coverage remained the same at 96.552% when pulling 3522354a45ea75a3c98acbcef2dff75fead380a3 on Hydrotoast:benchmarks into b2ea74801503396e40f8ecbc1910d684f1f1133f on btwardow:master.