Error while training on a different dataset.

[saharudra]

I was able to get the code to train on the provided datasets. I created the char map and followed the procedure based on other issues, mostly #13, to train on my own dataset but I am facing the following error.

see1: 1 windows (created Wed Apr  4 23:32:48 2018) [164x47] (attached)
ubuntu@ip-172-31-15-55:~$ cd src/see
ubuntu@ip-172-31-15-55:~/src/see$ python3 chainer/train_svhn.py --char-map ../datasets/BornDigital/borndigital_char_map.json -b 32 ../datasets/BornDigital/curriculum.json ../datasets/BornDigital/logs/ --b
lank-label 0 -g 0 -lr 0.0001
/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np
.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/usr/local/lib/python3.5/dist-packages/chainer/training/updaters/multiprocess_parallel_updater.py:131: UserWarning: optimizer.eps is changed to 1e-08 by MultiprocessParallelUpdater for new batch size.
  format(optimizer.eps))
Exception in main training loop: '109'
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/trainer.py", line 299, in run
    update()
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/updater.py", line 223, in update
    self.update_core()
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 206, in update_core
    loss = _calc_loss(self._master, batch)
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 235, in _calc_loss
    return model(*in_arrays)
  File "/home/ubuntu/src/see/chainer/utils/multi_accuracy_classifier.py", line 48, in __call__
    reported_accuracies = self.accfun(self.y, t)
  File "/home/ubuntu/src/see/chainer/metrics/svhn_softmax_metrics.py", line 54, in calc_accuracy
    label = "".join(map(self.label_to_char, label))
  File "/home/ubuntu/src/see/chainer/metrics/loss_metrics.py", line 181, in label_to_char
    return chr(self.char_map[str(label)])
Will finalize trainer extensions and updater before reraising the exception.
Traceback (most recent call last):
  File "chainer/train_svhn.py", line 257, in <module>
    trainer.run()
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/trainer.py", line 313, in run
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/trainer.py", line 299, in run
    update()
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/updater.py", line 223, in update
    self.update_core()
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 206, in update_core
    loss = _calc_loss(self._master, batch)
  File "/usr/local/lib/python3.5/dist-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 235, in _calc_loss
    return model(*in_arrays)
  File "/home/ubuntu/src/see/chainer/utils/multi_accuracy_classifier.py", line 48, in __call__
    reported_accuracies = self.accfun(self.y, t)
  File "/home/ubuntu/src/see/chainer/metrics/svhn_softmax_metrics.py", line 54, in calc_accuracy
    label = "".join(map(self.label_to_char, label))
  File "/home/ubuntu/src/see/chainer/metrics/loss_metrics.py", line 181, in label_to_char
    return chr(self.char_map[str(label)])
KeyError: '109'

Thanks

[Bartzi]

how many different characters do you want to distinguish? How many outputs does your last Linear layer have?

If those numbers are not the same you should get an error like that. It tells you that you predicted a class (in this case 109) that is not contained in the char_map.

[rohit12]

We are trying to distinguish between 57 + 1 blank character. The last Linear layer has 58 outputs. However, this error still occurs.

Initially, I thought that this might be due to incorrect char_map but I think I have used it correctly.

The following is the char_map that we are using. I think it is the correct format.

{  
   "0":122,
   "1":40,
   "2":121,
   "3":95,
   "4":58,
   "5":35,
   "6":163,
   "7":100,
   "8":106,
   "9":39,
   "10":34,
   "11":112,
   "12":56,
   "13":49,
   "14":41,
   "15":103,
   "16":42,
   "17":104,
   "18":38,
   "19":55,
   "20":53,
   "21":36,
   "22":52,
   "23":102,
   "24":48,
   "25":32,
   "26":116,
   "27":98,
   "28":118,
   "29":109,
   "30":37,
   "31":111,
   "32":114,
   "33":99,
   "34":110,
   "35":63,
   "36":107,
   "37":105,
   "38":54,
   "39":47,
   "40":120,
   "41":113,
   "42":50,
   "43":115,
   "44":33,
   "45":57,
   "46":64,
   "47":119,
   "48":8364,
   "49":46,
   "50":45,
   "51":117,
   "52":101,
   "53":231,
   "54":97,
   "55":174,
   "56":108,
   "57":51
}

[Bartzi]

yeah, that looks good and sounds right. Did you check your groundtruth? Do your labels contain numbers that are smaller than 57 everytime?

[saharudra]

We were able to get it to work. The problem was with the ground truth format.

Can you suggest some ways of making sure whether the ground truth alignment is correct? Currently we are thinking of just looking at validation output over training.

[Bartzi]

That's a very good question. I already had a look at the bboxes predicted by the model for every iteration of the training. This always gave me good intuition, whether there might be problems or not...

[saharudra]

Thanks for the suggestion.

I have written a medium draft on the paper and would love to hear any suggestions that you might have to improve it or make it clear.

https://medium.com/@rudrasaha1993/paper-summary-see-towards-semi-supervised-end-to-end-scene-text-recognition-7b7efbc9363a

[Bartzi]

that's a nice article. I like the article and think it is quite clear and concise.

[harshalcse]

how many different characters do you want to distinguish? How many outputs does your last Linear layer have?

If those numbers are not the same you should get an error like that. It tells you that you predicted a class (in this case 109) that is not contained in the char_map.

but our char map does not contains that class still why it is predicted and if our predicted class conains only alpha numeric labels then what changes need to do, please help me as I am completely new in neural network and Deep Learning stuff.