Add classes after classifier creation

[JakeAustwick]

Apologies in advance as my knowledge of Go is still somewhat limited, so this may be a naive question.

I want to expose the naive bayes classifying as a HTTP Web service, with both train and classify endpoints. I have no trouble with that, but I want the train endpoint to be able to accept new labels (labels that aren't currently in the classifier). Right now the labels are simply specified as consts and passed into the constructor. Can you think of the best way to add the ability to add labels at run-time?

[jbrukh]

Hi Jake, 

Great question! Let me think for a second, as I haven't looked at the bayesian code for a while. 

Jake 

— Sent from Mailbox

On Tue, Nov 4, 2014 at 2:13 PM, JakeAustwick notifications@github.com wrote:

Apologies in advance as my knowledge of Go is still somewhat limited, so this may be a naive question.

I want to expose the naive bayes classifying as a HTTP Web service, with both train and classify endpoints. I have no trouble with that, but I want the train endpoint to be able to accept new labels (labels that aren't currently in the classifier). Right now the labels are simply specified as consts and passed into the constructor. Can you think of the best way to add the ability to add labels at run-time?

Reply to this email directly or view it on GitHub: https://github.com/jbrukh/bayesian/issues/5

[jbrukh]

I believe all you would have to do is add a method to the Classifier struct that would look something like this (not tested):

func (c *Classifier) AddClass(class Class) {
    c.Classes = append(c.Classes, class)
    c.datas[class] = newClassData()
}

This will add the new class and subsequent calculations will take it into account. Note this is not particularly thread-safe if you're adding new classes asynchronously because the number of classes is a parameter to the calculations.

If you submit a pull request + unit test, I would be happy to merge the change. There's also some cleanup that can be done on the NewClassifier method once this method is added.

Thanks, Jake

[JakeAustwick]

The above code works, I tested it on our codebase - thanks for that. Are you still open to a patch even though you state you know it wouldn't be thread safe?

In our application I simply wrapped all of our interactions with the classifier with a mutex, but I'm guessing that would be undesirable to do inside the library itself?

[jbrukh]

IMO it would be best to leave thread-safety to the user, as it adds performance overhead, etc.

Just out of curiosity, what are you using it for? And how big is your data set? :)

[JakeAustwick]

We're using it to identify categories for uncategorized products based on their keywords / title.

As of right now, our dataset is < 10k training data items, and approx. different 250 labels.

I'll look at contributing the above patch today/tomorrow along with tests, as I haven't looked at testing in Go yet.

[jbrukh]

Cool!

I'm curious about the performance on that much data, as the library does everything in memory. It would be interesting to think about how to support larger sets. Also, does this approach to categorization work well?

As for testing, check out this page for basic Golang testing: https://golang.org/doc/code.html#Testing. I've also recently been using Ginkgo (http://onsi.github.io/ginkgo/) on other projects, which is similar to Rspec and works great -- highly recommend.

Thanks for your interest!

Jake

On Wed, Nov 5, 2014 at 1:10 PM, JakeAustwick notifications@github.com wrote:

We're using it to identify categories for uncategorized products based on their keywords / title.

As of right now, our dataset is < 10k training data items, and approx. different 250 labels.

I'll look at contributing the above patch today/tomorrow along with tests, as I haven't looked at testing in Go yet.

— Reply to this email directly or view it on GitHub https://github.com/jbrukh/bayesian/issues/5#issuecomment-61853906.

[tmaiaroto]

That's not a terrible amount of data to load into memory in my opinion. I've loaded 2.7 million items into a struct for an in memory geocoding package and it only used about 500MB of RAM. I've also done something similar for gender detection with a much smaller list. The performance is fast in memory and unless you're storing millions of items, the memory allocation should be quite small.

That said, 250 different classifications is quite a lot and I too would be curious about a benchmark in this case.