search

karpathy / char-rnn

Multi-layer Recurrent Neural Networks (LSTM, GRU, RNN) for character-level language models in Torch
11.62k stars 2.59k forks source link

Scoring Functionality #59

Open ryanleary opened 9 years ago

ryanleary commented 9 years ago

It would be nice to add in a third script score.lua which would take a trained model and an example (or many examples) and provide log probabilities of the character sequence according to the model.

I've started looking into adding this based on the sampling code, but have to learn Lua and Torch in the process. I imagine it would be a pretty simple addition for someone who knows what they're doing?

karpathy commented 9 years ago

I'd look at eval_split function in the main training code instead of the sampling script.

ryanleary commented 9 years ago

A few follow up questions:

  1. I see that we obtain a per-character log-likelihood as the output of the rnn: prediction = lst[#lst]. For scoring purposes, If the input sequence is 'hello', we would feed in 'h', get the LP for 'e' and add that to some accumulator. From there, forward 'e', get the LP for 'l', accumulate that, and so on. Is this correct?
  2. If the above is correct, how does one determine the initial distribution of probabilities for a given character at the start of a sequence.
  3. And finally, how does one reset the state of the RNN for scoring multiple sequences?

Thanks for your time!

JohnReid commented 9 years ago

@ryaneleary Did you ever get this scoring script working? I would be very interested to see/use it. @karpathy I saw your talk in Whitechapel the other Monday. Thanks for a great introduction to RNNs and LSTMs.

ryanleary commented 9 years ago

@JohnReid I did get it working. If you look at the closed PR (#61) you can see the implementation I used. It hasn't been rebased on any of the changes that have been going on in the main char-rnn code, so it may require some cleanup/modifications if you need it to run against models built by the latest version. Otherwise, it seemed to work fine for my task (scoring a given string against two different models and then compute a LLR).

JohnReid commented 9 years ago

@ryaneleary I saw this and I have a slightly modified version working now. Many thanks.

ryanleary commented 9 years ago

@JohnReid I've added you as a collaborator on my fork of the repo. If you made improvements to the script and wouldn't mind pushing them to a branch, I'd greatly appreciate it. My attention has been elsewhere recently, but I have definite plans to be using this in the near future.