Open raaaaaymond opened 6 months ago
Hi, I encountered same issues (working on https://github.com/harvardnlp/annotated-transformer/ ) Any updates on that?
Hi, I encountered same issues (working on https://github.com/harvardnlp/annotated-transformer/ ) Any updates on that?
I'm learning this repository too. And this issue happend when running:
train, val, test = datasets.Multi30k(language_pair=("de", "en"))
Inspired by processing datasets separately, this modified code can run successfully:
train = datasets.Multi30k(root='.data', split='train', language_pair=('de', 'en'))
val = datasets.Multi30k(root='.data', split='valid', language_pair=('de', 'en'))
But if you add test
, the error will occur, for reasons I don't know.
The package version is following:
pytorch 2.1.2 py3.11_cuda12.1_cudnn8_0 pytorch
pytorch-cuda 12.1 hde6ce7c_5 pytorch
pytorch-mutex 1.0 cuda pytorch
torchaudio 2.1.2 pypi_0 pypi
torchdata 0.7.1 py311 pytorch
torchtext 0.16.2 py311 pytorch
torchvision 0.16.2 pypi_0 pypi
The error occurred because the original server went down, so the download link for the Multi30k dataset was temporarily modified. For example, the download link for the test set is: https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz, This compressed file contains .test.*
and test.*
files( represents en, de, fr). However, the _filter_fn function filters files based on whether the filename contains: `test., which also matches
.test.files. The
.test.` files contain illegal utf-8 characters, causing an error when reading the file.
Possible solutions are:
/test.*
(with an additional slash) instead of test.*
, eg:return f"/{_PREFIX[split]}.{language_pair[i]}" in x[0]
.test.*
files to the torch cache directory: ~/.cache/torch/text/datasets/Multi30k/
.The test set is still not working!
I'm using the following version:
torch 2.2.2 pypi_0 pypi
torchdata 0.7.1 pypi_0 pypi
torchtext 0.17.2 pypi_0 pypi
It constantly throws an error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 37: invalid start byte.
🐛 Bug
Describe the bug A clear and concise description of what the bug is.
Getting the following error while attempting to iterate through the testing set of (the 3rd thing returned by)
torchtext.datasets.Multi30k(language_pair=("de", "en"))
:UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 37: invalid start byte
To Reproduce Steps to reproduce the behavior:
torch torchaudio torchdata torchtext torchvision
.UnicodeDecodeError
.Expected behavior A clear and concise description of what you expected to happen.
There should be no error, but a print out of the data. For example, if instead of
for thing in testing
, we didfor thing in validation
orfor thing in training
, then everything works as expected.Screenshots If applicable, add screenshots to help explain your problem.
Not necessary; just observe the error.
Environment
Please copy and paste the output from our environment collection script (or fill out the checklist below manually).
Additional context Add any other context about the problem here.