search

louismullie / treat

Natural language processing framework for Ruby.
Other
1.37k stars 128 forks source link

tf*idf: Why don't you normalize to maximum term count for document? #22

Open jpmckinney opened 12 years ago

jpmckinney commented 12 years ago

Treat (and the tf-idf and similarity gems) all normalize tf to the number of terms in the document: https://github.com/louismullie/treat/blob/master/lib/treat/workers/extractors/tf_idf/native.rb#L78

We normalize so that (1) long and short documents have comparable tf weights and (2) documents with large vocabularies and those with small vocabularies have comparable tf weights.

Normalizing to the number of terms in a document only fixes (1). Normalizing to the maximum term count (which is all I've ever seen in the literature) fixes both (1) and (2).

louismullie commented 12 years ago

It should be included in the next release of the gem.

jpmckinney commented 12 years ago

I wrote my own gem in the meantime: https://github.com/opennorth/tf-idf-similarity Perhaps you'd like to re-use it?

louismullie commented 12 years ago

Sweet. I will! My philosophy is really to keep as much logic code as possible separate from the Treat core, so this is definitely the kind of contribution I welcome. My naïve implementation was horribly inefficient anyway.

Thanks, Louis

louismullie commented 12 years ago

I am interested in using your library to replace the current tf_idf workers, but my requirements would be as follows:

1 - We need to be able to input documents that are already tokenized. 2 - We need to be able to easily access tf*idf scores for a given word in a given document.

From what I understand, the gem does not provide a public interface to perform either of these tasks.

Are you interested in coding up the public functions so I can use it directly? Or should I add it to my (long) to-do list for Treat?

jpmckinney commented 12 years ago
  1. I assume the tokenized documents would be arrays? You can now do that by passing :tokens argument when initializing a document, eg. Document.new("Lorem ipsum", :tokens => ["Lorem", "ipsum"])
  2. If you have a collections of documents, you can now get the tf*idf for a particular document and term with collection.tfidf(document, term). Previously you'd have to multiply the idf and tf yourself, eg:
collection.idf(term) * collection.tf(document, term)