The command line help indicates that gzipped input files are supported. However, if a gzipped training data file or validation data file is given, training fails with UnicodeDecodeError.
File "/l/sgronroo/scratch/theanopy3/bin/theanolm", line 12, in
exec(compile(open(file).read(), file, 'exec'))
File "/l/sgronroo/scratch/theanopy3/theanolm/bin/theanolm", line 46, in
main()
File "/l/sgronroo/scratch/theanopy3/theanolm/bin/theanolm", line 41, in main
args.command_function(args)
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/commands/train.py", line 303, in train
trainer.train()
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/trainers/basictrainer.py", line 132, in train
self._validate(perplexity)
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/trainers/localstatisticstrainer.py", line 57, in _validate
perplexity = self.scorer.compute_perplexity(self.validation_iter)
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/scoring/textscorer.py", line 130, in compute_perplexity
for wordids, , mask in batch_iter:
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/iterators/batchiterator.py", line 93, in next
sequence = self._read_sequence()
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/iterators/batchiterator.py", line 168, in _read_sequence
for word in utterance_from_line(line)]
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/iterators/batchiterator.py", line 20, in utterance_from_line
line = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
It appears that the problem is caused by the mmap access to the file (SentencePointers in iterators/shufflingbatchiterator:66) failing for gzipped files. The transparent unzipping (implemented in TextFileType filetypes.py:95) has no effect when using mmap.
The command line help indicates that gzipped input files are supported. However, if a gzipped training data file or validation data file is given, training fails with UnicodeDecodeError.
It appears that the problem is caused by the mmap access to the file (SentencePointers in iterators/shufflingbatchiterator:66) failing for gzipped files. The transparent unzipping (implemented in TextFileType filetypes.py:95) has no effect when using mmap.