search

bnosac / ruimtehol

R package to Embed All the Things! using StarSpace
Mozilla Public License 2.0
99 stars 13 forks source link

starspace test mode #20

Closed jiunsiew closed 4 years ago

jiunsiew commented 5 years ago

Hi there. Thanks for developing this package. It's really great work.

I'm trying to figure out how we might use the starspace train functionality that's available with Starspace from within the R package.

To be more specific, I've created a model using model <- embed_tagspace(...) and would now like to run predictions on a hold out test set. I know it's possible to use the predict function to do this (have already done so) but in the starspace example shell scripts, they have the ability to do starspace test -model ... which spits out some automated test metrics. Is it possible to use this starspace functionality directly from within the R environment?

I can see from https://github.com/bnosac/ruimtehol/blob/master/src/rcpp_textspace.cpp that the test input argument is being created. I just can't quite figure out how to pass this to the embed_ or starspace functions. Any help would be greatly appreciated.

Thank you.

jwijffels commented 5 years ago

You can use predict(model, newdata = ....................., type = "labels") to get predictions. It will give you cosine or dot product similarity measures between your text and the categories. There are really many R packages which give classification metrics, I think the evaluation metrics in Starspace are not so valuable. Look at R packages on CRAN which give you standard classification metrics or multi-label classification metrics. If you really want to use the starspace test mode, you can use what is coded here https://github.com/bnosac/ruimtehol/blob/master/src/rcpp_textspace.cpp#L377 and call that as follows: ruimtehol:::textspace_evaluate(model$model, testFile = "path-to-file-containing-testdata", basedoc = "this is really optional", predictionFile = "path-to-output-file", int K = 5) but at your own discretion (might crash, I have been testing this out but forgot myself how to call this functionality because you should create your own evaluation functionality on your downstream task).

jiunsiew commented 5 years ago

Thanks. I'll give that a try. I was using the predict before. Was just wondering if the Starspace test method was available.

Thanks for the quick response.

jwijffels commented 4 years ago

Closing as answer was given. Feel free to reopen if needed.