Troubleshooting invalid one-block filenames

sosedoff commented 2 years ago

Hit an edge case where merger was not picking up any one-block files from a local store:

0005298257-20210226T082357.4-F5C5F498-370E7AAF-dans-mbp.lan.dbin.zst
0005298258-20210226T082404.2-6BF72A71-F5C5F498-dans-mbp.lan.dbin.zst
0005298259-20210226T082411.8-8D7A105E-6BF72A71-dans-mbp.lan.dbin.zst
0005298260-20210226T082418.8-2D41ABBE-8D7A105E-dans-mbp.lan.dbin.zst
0005298261-20210226T082426.0-6C9900DE-2D41ABBE-dans-mbp.lan.dbin.zst
0005298262-20210226T082432.7-119B2B90-6C9900DE-dans-mbp.lan.dbin.zst
0005298263-20210226T082439.6-6EA67951-119B2B90-dans-mbp.lan.dbin.zst
0005298264-20210226T082446.3-AE7A18B0-6EA67951-dans-mbp.lan.dbin.zst
0005298265-20210226T082453.3-66FE6DDD-AE7A18B0-dans-mbp.lan.dbin.zst
0005298266-20210226T082500.2-6D98ADDF-66FE6DDD-dans-mbp.lan.dbin.zst

Merger performed its run and logged the following:

{
  "level": "info",
  "time": "2021-12-01T17:54:30.647-0600",
  "msg": "retrieved list of files",
  "seenblock_low_boundary": 0,
  "bundle_lower_block": 5298200,
  "seen_files_count": 0,
  "too_old_files_count": 0,
  "good_files_count": 0,
  "canonical_good_files_count": 0
}

I thought the start block number was off so i've changed a few params with no change in behavior. Dropped into the merger code from there and found this part (error logging is skipped):

https://github.com/streamingfast/merger/blob/ca8b1a40ae2a17c371c43ad7d0163d8fe5ca6180/merger.go#L349

the core of the issue is filename check in here: https://github.com/streamingfast/merger/blob/ca8b1a40ae2a17c371c43ad7d0163d8fe5ca6180/paths.go#L36

For testing purposes i set the one block file suffix to hostname, which is dans-mbp.lan

Maybe we should validate the suffix in the mindreader?

abourget commented 2 years ago

I think the issue was with the . in the hostname. Can you confirm?

sosedoff commented 2 years ago

Sure no prob, will confirm again

matthewdarwin commented 2 years ago

I remember someone else running into a similar problem a few months ago.

streamingfast / merger

Troubleshooting invalid one-block filenames #4