Open stephbuon opened 2 years ago
@EliasLMann would you be interested in attempting this or nah?
I showed you the TF-IDF measures that we have throughout. We would like more "transparent" measures implemented like log likelihood.
Ideally I would be able to implement log likelihood in real time, like the TF-IDF function.
Yes, definitely. I'll give it a shot starting Sunday night.
Thanks, @EliasLMann!
Here's some background: I want to implement this in the same way that I use TF-IDF for collocates or speaker comparison (so the user will be able to select Log Likelihood from the drop down).
I think it should take a data frame as input, and return another data frame, but with log likelihood scores.
A guess a key thing to check before wrapping this, however, is that this implementation of LL is actually fast enough to run in real time on our data.
R has a few log likelihood implementations, but they're all exceptionally slow and they would be unrealistic in a web app.
I probably need a lower level, C++ implementation of log likelihood.
I found this C++ package (maybe there are better options): http://www.cahillsoftware.com/media/af21843f135214fffff850fffffe417.html
Should I make an R wrapper for it using Rcpp?
Here's the thing though: I'm not sure how to apply the above package to a data frame -- nor do I really understand how to measure log likelihood correctly, so I am afraid I am going to mess something up.
Ideally I would find the right person for this job.