Closed MBrouns closed 10 years ago
There are some undefined methods in Utilities like getMostFrequentWords and getWordFrequency. Are these the ones yet to be implemented or am I missing something?
Is the classifier4j jar path correctly set in your classpath?
On Fri, Mar 21, 2014 at 10:10 AM, Friso Abcouwer notifications@github.comwrote:
There are some undefined methods in Utilities like getMostFrequentWords and getWordFrequency. Are these the ones yet to be implemented or am I missing something?
Reply to this email directly or view it on GitHubhttps://github.com/yetti4/crowd-summary/pull/13#issuecomment-38258833 .
It is. The missing methods are in javax.swing.text.Utilities anyway
Ah, -import net.sf.classifier4J.Utilities. Never mind!
Oh what I remembered by the way, we don't really need our own implementation for the tf, I think we can use the Utilities.getwordfrequency for that. We can then iterate over that list and calculate the idf for those terms. Advantage is that it has a default stop word ignore list so the list will be a lot smaller than all the words in the document. Could shave some time of the calculation
On Fri, Mar 21, 2014 at 10:15 AM, Friso Abcouwer notifications@github.comwrote:
Ah, -import net.sf.classifier4J.Utilities. Never mind!
Reply to this email directly or view it on GitHubhttps://github.com/yetti4/crowd-summary/pull/13#issuecomment-38259104 .
:+1: I'll check it out
I improved the TF/IDF calculation, but a lot of it is still clunky. I expect I'll have time to look at it on Tuesday, otherwise we'll just continue working on it on Wednesday.
I made a small start on implementing the tf-idf for getting a list of most important words in the document. @fabcouwer do you think you can further expand this implementation?
I've taken a quick look at the classifiers that classifier4j offers and it seems that the VectorClassifier should be usable for us to implement for ranking the quality of sentences that are to be put into the database. The problem with this method is that you cannot incrementally add training data which means that the entire training has to be done every single time. However, the performance is better then their normal bayes classifier.
Implementation of the classifier seems like quite some work so maybe it's best if we take a look at it together next week. In the meantime though it's quite important to get some training data so we can actually start using it when we want to implement. I'm not sure if @fabcouwer has time for this but maybe @timsweep can join him?