Cannot Reproduce on IAM-OnDB

[bhavinjawade]

Hi,

Using the checkpoint provided in readme, we are getting extremely low numbers on IAM-onDB test_t and test_f testing sets (even on the validate set). Can you verify if you can reproduce the numbers reported in your paper with that checkpoint and the code on the github. Also can you provide the command-line script that you used to get those numbers?

Thanks, Bhavin

[jungomi]

Hello,

Thank you for trying it out. It turns out that there was a bug in the GT conversion script, where the point index and stroke index were accidentally swapped (was the result of refactoring for the public release). This is now fixed, and you can just run the conversion again and it should work.

After the fix, the following command should give the correct results:

python evaluate_model.py -m model-character-queries.pt -d "Test Set T=data/converted/iam/testset_t_segmented.tsv" "Test Set F=data/converted/iam/testset_f_segmented.tsv" -o results/repro

Note: I used NAME=PATH to make the names nicer, but the path itself is enough (the quotes are necessary to have spaces in the name, otherwise it's parsed as a new argument)

Result:	Model / Dataset	Test Set T	Test Set F
model-character-queries.pt	92.42	95.06

There is a very slight deviation from the numbers in the paper, that is most likely attributed to the batch size, which defaults to 32 in the evaluation script, and I believe we just maxed it out for the GPU we were using.

[bhavinjawade]

Thanks a lot @jungomi. Its a great paper!

[bhavinjawade]

Could you please also provide the chars.tsv file that the code requires and combined dataset tsv files (train.tsv)?

Thanks and Regards, Bhavin Jawade

[jungomi]

Sure.

chars.txt I had to change it to a .txt file because GitHub does not allow uploading files with a .tsv extension here. It doesn't matter anyway.

For the combined train set, there is nothing special about it, just concatenate the two train sets (IAM-OnDB and VNOnDB). By using a TSV file to list the files for the dataset makes it very simple to add/remove files without having to move them around. This really depends on where your data is. If you have the two train sets in data/converted/iam and data/converted/vnondb respectively, you can create a data/converted/train-combined.tsv and then prefix the paths of the respective training sets with iam/ or vnondb/. So that would look something like that:

iam/trainset_segmented/l04-397z-05.xml.json
iam/trainset_segmented/f01-075-02.xml.json
...
vnondb/InkData_word_train_segmented/20140925_0043_5770_3.inkml_335.json
vnondb/InkData_word_train_segmented/20140603_0033_KQBDVN.inkml_25.json
...

[sahanaranga]

Hi,

Would it be possible to share stats.pt so that we may train the model with the checkpoint provided?

Thanks, Sahana

[jungomi]

Hey @sahanaranga, the stats.pt file isn't really necessary to train a model from a checkpoint, it just tracks the results of the training. The only difference it makes is when continuing the training, because that resumes the learning rate at the step it stopped, and to determine whether a new best checkpoint was found.

I do realise that the code technically requires this (for any newer codebases I have separated that properly, along other much needed improvements), but the simplest is to just remove the loading of the stats.pt entirely:

https://github.com/jungomi/character-queries/blob/2c4ca3d59ea787cd612ba57852cc0d8233028a72/train.py#L885

Without this line it should be fine to train from the given checkpoint, which must be in the directory that you specify to -c/--checkpoint as model.pt.

[sahanaranga]

Thank you!