search

danpovey / lilcom

Small compression utility
Other
33 stars 10 forks source link

ValueError: lilcom: Length of string was too short #47

Closed shanguanma closed 1 year ago

shanguanma commented 1 year ago

I am using lilcom compress and decompress the numpy data, however I got the below error:

WARNING:root:len(f.read()) is 277089
WARNING:root:len(f.read()) is 906037
Traceback (most recent call last):
  File "train.py", line 492, in <module>
    main()
  File "train.py", line 488, in main
WARNING:root:len(f.read()) is 777463
WARNING:root:len(f.read()) is 558017
    train(a.local_rank, cfg, a)
  File "train.py", line 221, in train
    for i, batch in pb:
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/deepspeed/runtime/dataloader.py", line 90, in __next__
    return next(self.data)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/deepspeed/runtime/dataloader.py", line 108, in <genexpr>
WARNING:root:len(f.read()) is 231040
    self.data = (x for x in self.dataloader)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
    return self._process_data(data)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mntnfs/lee_data1/maduo/w_2022/librispeech/transfusion-asr/transfusion/dataset.py", line 58, in __getitem__
    arr = lilcom.decompress(f.read())
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 99, in decompress
    shape = lilcom_extension.get_float_matrix_shape(byte_string)
ValueError: lilcom: Length of string was too short

Could you help me？

shanguanma commented 1 year ago

@danpovey, @csukuangfj

csukuangfj commented 1 year ago

File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 99, in decompress shape = lilcom_extension.get_float_matrix_shape(byte_string)

Please use pdb to get the length of byte_string.

shanguanma commented 1 year ago

@csukuangfj ,sorry for replying so late, I follow your suggestion, I used gpd instead of pdb, the log is as follows:

2022-11-09 18:05:46,484 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/64/76974/64-76974-0002.llc,byte_string is 1038371
2022-11-09 18:05:46,484 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/1100/135264/1100-135264-0001.llc,byte_string is 154094
2022-11-09 18:05:46,485 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/7609/39979/7609-39979-0049.llc,byte_string is 964635
2022-11-09 18:05:46,485 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/5733/55099/5733-55099-0044.llc,byte_string is 398394
2022-11-09 18:05:46,486 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/6981/70843/6981-70843-0109.llc,byte_string is 965834
2022-11-09 18:05:46,486 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/4915/113765/4915-113765-0038.llc,byte_string is 902560
2022-11-09 18:05:46,487 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/7481/101276/7481-101276-0060.llc,byte_string is 776831
Traceback (most recent call last):
  File "./train_with_torch.py", line 813, in <module>
    main()
  File "./train_with_torch.py", line 806, in main
    run(rank=0, world_size=1, args=args)
  File "./train_with_torch.py", line 765, in run
    train_one_epoch(
  File "./train_with_torch.py", line 587, in train_one_epoch
    for batch_idx, batch in enumerate(train_dl):
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/8176/115046/8176-115046-0031.llc,byte_string is 1001308
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/4019/8895/4019-8895-0053.llc,byte_string is 983517
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/2544/27475/2544-27475-0016.llc,byte_string is 898850
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/8215/275211/8215-275211-0057.llc,byte_string is 547630
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/724/123284/724-123284-0012.llc,byte_string is 965144
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/727/124443/727-124443-0147.llc,byte_string is 766136
2022-11-09 18:05:46,488 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-100/250/142276/250-142276-0022.llc,byte_string is 874249
2022-11-09 18:05:46,489 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-clean-360/4222/187006/4222-187006-0015.llc,byte_string is 1051919
    data = self._next_data()
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
    return self._process_data(data)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
2022-11-09 18:05:46,491 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/1049/133454/1049-133454-0055.llc,byte_string is 898126
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mntnfs/lee_data1/maduo/w_2022/librispeech/transfusion-asr/transfusion/dataset.py", line 62, in __getitem__
    arr = lilcom.decompress(f.read())
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 99, in decompress
    shape = lilcom_extension.get_float_matrix_shape(byte_string)
ValueError: lilcom: Length of string was too short

2022-11-09 18:05:46,492 WARNING [dataset.py:60] row.wavlm_path data/wavlm_feat/train-other-500/3691/178109/3691-178109-0026.llc,byte_string is 211509
[Thread 0x1554b2e18700 (LWP 4060367) exited]
[Thread 0x15545583f700 (LWP 4060368) exited]

I think the above utterance is ok, not too short.

shanguanma commented 1 year ago

Could you add the length of the string in lilcom library, the ValueError hasn't been a big help to debug.

shanguanma commented 1 year ago 
>>> import lilcom
>>> f = open("data/wavlm_feat/train-clean-100/103/1240/103-1240-0000.llc",'rb')
>>> len(f.read())
909177
>>> arr = lilcom.decompress(f.read())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 99, in decompress
    shape = lilcom_extension.get_float_matrix_shape(byte_string)
ValueError: lilcom: Length of string was too short
>>> len(f.read())
0
>>> with open("data/wavlm_feat/train-clean-100/103/1240/103-1240-0000.llc",'rb')as f:
...     print(len(f.read()))
...     arr = lilcom.decompress(f.read())
...     print(arr)
...
909177
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/mntnfs/lee_data1/maduo/anaconda3/envs/transfusion-asr/lib/python3.8/site-packages/lilcom/lilcom_interface.py", line 99, in decompress
    shape = lilcom_extension.get_float_matrix_shape(byte_string)
ValueError: lilcom: Length of string was too short
csukuangfj commented 1 year ago 
>>> len(f.read())
909177
>>> arr = lilcom.decompress(f.read())

Note: len(f.read()) has already set EOF for f.

If you invoke f.read() again, it returns nothing.

shanguanma commented 1 year ago

Yes, I found the issue just now. thanks a lot