Fixed transition to new words skipping probabilities

[ChWick]

I got several problems on rather trivial dictionaries and charsets (see below):

• Without pull-request:

  Greedy-Decoder:  # 4 3 5 4 2 3 5 6 7 6 7 5 4 5 4 3 5 . 2 3
  Token-Passing:   # 4 3 5 4 2 5 6 7 7 5 4 5 4 3 . 2 3

• With pull-request (as expected):

  Greedy-Decoder:  # 4 3 5 4 2 3 5 6 7 6 7 5 4 5 4 3 5 . 2 3
  Token-Passing:   # 4 3 5 4 2 3 5 6 7 6 7 5 4 5 4 3 5 . 2 3

• Test code (copy the snipped in TokenPassing.py and use all files in the appendix)

  
  with open("charset.txt") as f:
  classes = f.read().split("\n")
  with open("dictionary.txt") as f:
  dictionary = f.read().split("\n")
  probabilities = np.load("propabilities.npy")

greedy

from itertools import groupby best_path = np.argmax(probabilities, axis=1) blank_idx = 0 best_charscollapsed = [classes[k] for k, in groupby(best_path) if k != blank_idx] res = ' '.join(best_chars_collapsed) print("Greedy-Decoder: ", res)

token passing

tp = ctcTokenPassing(probabilities, classes, dictionary, blankIdx=blank_idx) print("Token-Passing: ", tp)

import matplotlib.pyplot as plt plt.imshow(np.log(probabilities)) plt.show()

The problem is that the current code allows to skip a single output on the first transition (0->1,2) since `toks.get(wIdx, s - 1, t - 1)` is chosen with does not include the probability at `t`. Using `toks.get(wIdx, s - 1, t)` instead fixes this, because this includes the emission probability for `blank`.

There might be another crucial issue if a there is no blank between two words. I had no time to check this, though.

Appendix:
![probabilities](https://user-images.githubusercontent.com/6162410/70219628-7026e180-1745-11ea-9ded-5d33dd1704c4.png)

[charset.txt](https://github.com/githubharald/CTCDecoder/files/3925859/charset.txt)
[dictionary.txt](https://github.com/githubharald/CTCDecoder/files/3925860/dictionary.txt)
[propabilities.zip](https://github.com/githubharald/CTCDecoder/files/3925862/propabilities.zip)

[githubharald]

thank you for the detailed analysis. I hope that I will soon have time to look at it.

Best, Harald

[githubharald]

still didn't had the time to look at this in more detail. However, your analysis seems reasonable, so I merged it. Thanks for your contribution :+1: . For reasons of consistency I changed the blank index back to the (fixed) last position in the RNN output.