Support non-ascii languages in file-level tokenizer and fix tests

[danhper]

I was getting encoding errors in the file-level tokenizer when trying to tokenize files with non ascii characters. Tests were broken so I fixed them and added a regression test.