Vdash failed to load invalid UTF-8

willief commented 2 years ago

Crashed after attempting to reload without the -i flag

willie@gagarin:~$ vdash  ~/.safe/node/baby-fleming-nodes/sn-node-*/sn_node.log 
Loading 12 files...
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-10/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-11/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-12/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-2/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-3/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-4/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-5/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-6/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-7/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-8/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-9/sn_node.log
file: /home/willie/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log
stream did not contain valid UTF-8

happybeing commented 2 years ago

Thanks Willie, I'd not seen this for a while so was hoping a MaidSafe change has fixed it but no :-(

If you have time can you see if you can isolate this to a particular logfile? It's saying there's an invalid character in one of the files (though it may be lying).

If it appears to be only in one logfile we may be able to track this down (hint) by successively chopping the logfile in half to identify a particular line that causes it to crash. Time consuming work but repeatedly halving is quite efficient (lookup bisection algorithm!).

willief commented 2 years ago

Yes - I have more to add to this and other vdash niggles. The log files have all been saved but a quick grep for non-UTF-8 chars found nothing. Thanks for the hint .
The genesis node was the one that caused the crash(es). Originally I was running with 20-off 10GB vaults and ran out of memory on a 32GB RAM box with the same again in swap trying to put 2.8GB of photos but this was repeated after another reboot and a smaller bite of the cherry. So I was thinking its maybe down to instability under very high load. I saw a peak 1min load of 5000+.

happybeing commented 2 years ago

I think there are two likely causes:

the logs contain an invalid character. That would be a problem in sn_node. It's easier to check (by reloading the logs into vdash) so best to look for that first.
a bug in the library which is reading the logs, maybe a timing issue (as it's quite rare this happens). It didn't used to happen and that library has been updated, so it's a key suspect. This would be much harder to prove and solve so ideally we'd need to find a way to reproduce it consistently but that looks unlikely for now.

willief commented 2 years ago

I think we are on option 2 here. I just loaded all the saved logs into vdash and it loaded OK each time. It must have been something transient , a bug as you mention or possibly instability as I have never run this box so hard before - its only a few months old. I'll keep an eye out for anything similar. Possibly close this one now and refer back if need be?

happybeing commented 2 years ago

Thanks Willie that's helpful. Next thing is for me to raise an issue with linemux unless I can find something there about this already.

willief commented 2 years ago

I got that error again. However seconds later the big box ran out of memory entirely and froze. So this is from my phone. So we can probably discount this last occurrence.

On Tue, 15 Mar 2022, 16:21 happybeing, @.***> wrote:

Thanks Willie that's helpful. Next thing is for me to raise an issue with linemux unless I can find something there about this already.

— Reply to this email directly, view it on GitHub https://github.com/happybeing/vdash/issues/10#issuecomment-1068185276, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACHW3ORWTHDKNV74OH3FALVAC2JLANCNFSM5QXHGKOQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

happybeing / vdash

Vdash failed to load invalid UTF-8 #10