Closed makobot-sh closed 4 months ago
engine.Compiler.OptimiseInferenceCode = false;
and it worked. So it seems there is a bug in how Infer.NET is optimizing the inference code. Thanks for bringing this to my attention.Thank you so much for the response! Turning off code optimization improved the estimatations in both cases, and putting the if block around the whole model has definitely improved the geometric mean of the evidence for the 100 observations case (which is now 0.65, calculated $exp(\frac{log(evidence)}{numObs})$ ).
For the 20k observations case, however, the geometric mean evidence (and the evidence itself) is still 0. Do you have any idea why this could be? (Here's the amended code, also a fork with the code to reproduce on branch tutorial_prints
in case that's more convenient, and a diff against the original infer repo code)
On a different note, would the bug affect the full Matchbox Recommender implementation as well? How can we change said configuration for it if so?
It seems that toggling OptimiseInferenceCode
doesn't fix everything, it just makes some of the estimates better. I will look into a better solution. The full Matchbox Recommender implementation is not affected.
I see, if you do please let me know! It would be nice to get the evidence for a trained Matchbox Recommender, but as far as I could tell it's not possible. Thanks for everything!
This is fixed by #457
Thank you Tom! The tutorial works great now for both values of largeData. I was playing around with it a bit more and found that with bigData=true and 5+ traits instead of 2, the geometric mean of the evidence drops back down to 0 (and the "Evidence is too low" exception triggers). Is this expected behaviour? With bigData=false, 5 traits seem to work well (the geometric mean of the evidence is below randomness, but this could be attributed to the low number of observations).
Damping is required for large numbers of traits. The attached PR shows how to do this.
Hi everyone, I've been trying to get the evidence for the model from the Recommender System Tutorial in the docs but I haven't been able to figure out how. I tried using mixture modelling as described in Computing model evidence for model selection but the resulting evidence has been very small (near 0). I'm not sure I'm doing things right so if anyone could give me feedback on the code I'd appreciate it enormously. Here is what I did (look at the
RecommenderTutorialFromRepository.Evidence
function).Despite using the following arguments for the data generation:
As an aside, I haven't been able to reproduce the results at the end of the tutorial (20000 observations instead of 100). The arguments for that one are the following (you can copy-paste onto my code to test, at RecommenderTutorialFromRepository.cs:17):
The estimated item traits don't match the truth or the tutorial's results at all. Instead, I'm getting the following estimations:
Any ideas why this could be?