search

lnresearch / topology

Data about the past and current structure of the Lightning Network
MIT License
82 stars 15 forks source link

ValueError: Error reading dataset at 2552794999: incomplete read of 138 bytes, only got 136 bytes #12

Closed nymkappa closed 2 years ago

nymkappa commented 2 years ago

Hi, I'm testing the latest backup file gossip-20220719.gsp but it seems to be corrupted?

Replaying gossip messages: 17683809it [01:15, 235210.38it/s]
Traceback (most recent call last):
  File "/opt/homebrew/bin/lntopo-cli", line 8, in <module>
    sys.exit(cli())
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/lntopo/timemachine.py", line 39, in restore
    for m in tqdm(dataset, desc="Replaying gossip messages"):
  File "/opt/homebrew/lib/python3.9/site-packages/tqdm/std.py", line 1165, in __iter__
    for obj in iterable:
  File "/opt/homebrew/lib/python3.9/site-packages/lntopo/common.py", line 32, in __next__
    raise ValueError(
ValueError: Error reading dataset at 2552794999: incomplete read of 138 bytes, only got 136 bytes

Related code: https://github.com/lnresearch/topology/blob/main/lntopo/common.py#L32

cdecker commented 2 years ago

Indeed it appears as if the dataset is truncated, and the last message is missing two bytes.

guillaez commented 2 years ago

Hi, I've also tested your precious data, but the whole 'node_announcement' messages are truncated too. I guess some 'channel_update' messages seem to be also truncated. Could you please check and fix the dataset?

cdecker commented 2 years ago

The old dataset has been pulled and a new one has been added instead :+1: