search

k2-fsa / snowfall

Moved to https://github.com/k2-fsa/icefall
Apache License 2.0
143 stars 42 forks source link

error when PytorchStreamReader failed reading file #186

Open yujunlhz opened 3 years ago

yujunlhz commented 3 years ago

python3 ./mmi_bigram_train.py

World size: 1 Rank: 0 Traceback (most recent call last): File "./mmi_bigram_train.py", line 502, in main() File "./mmi_bigram_train.py", line 280, in main lexicon = Lexicon(lang_dir) File "/tmp/lingvo/snowfall/build/lib/snowfall/lexicon.py", line 21, in init L_inv = k2.Fsa.from_dict(torch.load(self.lang_dir / 'Linv.pt')) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 592, in load return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 851, in _load result = unpickler.load() File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 843, in persistent_load load_tensor(data_type, size, key, _maybe_decode_ascii(location)) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 831, in load_tensor storage = zip_file.get_storage_from_record(name, size, dtype).storage() RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading file data/57885792: invalid header or archive is corrupted

k2 0.3.3+cu102.dev20210507 kaldialign 0.1.7 keyring 23.0.1 lhotse 0.6.0 lilcom 1.1.1 Markdown 3.3.4 numpy 1.20.2 torch 1.8.1+cu102 torchaudio 0.8.1

what might be the culprit? Thanks

danpovey commented 3 years ago

I'm not sure what would cause this, but removing Linv.pt should work around it.

On Fri, May 7, 2021 at 11:18 PM yujunlhz @.***> wrote:

python3 ./mmi_bigram_train.py

World size: 1 Rank: 0 Traceback (most recent call last): File "./mmi_bigram_train.py", line 502, in main() File "./mmi_bigram_train.py", line 280, in main lexicon = Lexicon(lang_dir) File "/tmp/lingvo/snowfall/build/lib/snowfall/lexicon.py", line 21, in init L_inv = k2.Fsa.from_dict(torch.load(self.lang_dir / 'Linv.pt')) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 592, in load return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 851, in _load result = unpickler.load() File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 843, in persistent_load load_tensor(data_type, size, key, _maybe_decode_ascii(location)) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 831, in load_tensor storage = zip_file.get_storage_from_record(name, size, dtype).storage() RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading file data/57885792: invalid header or archive is corrupted

k2 0.3.3+cu102.dev20210507 kaldialign 0.1.7 keyring 23.0.1 lhotse 0.6.0 lilcom 1.1.1 Markdown 3.3.4 numpy 1.20.2 torch 1.8.1+cu102 torchaudio 0.8.1

what might be the culprit? Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/186, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO5A5IKZKQLR3VZC2BLTMQADZANCNFSM44KLCHLQ .

yujunlhz commented 3 years ago

it turn out that Linv.pt was generated with a another version. thanks