Add support for UTF-16 character encoding

[denniskhk]

This utility is extremely useful - I used to have to manually parse and enter the info into a spreadsheet by hand :((

This works great for logs created by the Chia UI logger. However, I have recently started to plot using the CLI. I pipe the output into a log via Powershell tee command. This produces a similar file BUT one major difference is that it does not have the entire block at the top - it starts with the "Starting plotting progress into temporary dirs: xxx" line onwards. I've attached a sample log file. _202105312205.log

This causes the utility to just crash out with no output. Perhaps it would be great if the utility could be modified so it'll skip missing info (treat as blank) and move on to parse the remaining info?

Thanks again for this useful time-saving tool!

[Pengor]

Hi there, unfortunately I was not able to reproduce your issue, on my PC I was able to parse the sample file you attached without any problems. The first line that Chia Log Analysis looks for is Starting plotting progress into temporary dirs: dir_1 dir_2 so I would expect the format to work just fine. Could you share these additional details and I can help troubleshoot why it's not working for you?

• Which operating system are you using?
• How did you format the command?
• What is the filepath that the log file resides in?

[denniskhk]

Dear Drew,

Thank you very much for the rapid response. As requested:-

Operating system: Windows 10 Pro x64 Command format: none, just double-clicked the exe which then opened up a command prompt window and ran the program Filepath: various. I have tried moving the log files to see if there were issues with path length etc, from the original C:\Users\Plotter\Desktop\pslogs to even just C:\pslogs. Same problem.

In addition, here are my observations:-

1) Works without problems with log files generated by the chia UI plotter - I copied 1 such file into the folder with all the powershell plotter logs, and the analysis app successfully parsed just that file. Same filepath as the other logs, and in fact the UI plotter logs have way longer filenames

2) The error returned is "no relevant data found in file" - I have attached a screenshot, which also shows some information in reply to your queries.

Thank you and best regards,

Dennis

On Tue, 1 Jun 2021 at 23:01, Drew M Johnson @.***> wrote:

Hi there, unfortunately I was not able to reproduce your issue, on my PC I was able to parse the sample file you attached without any problems. The first line that Chia Log Analysis looks for is Starting plotting progress into temporary dirs: dir_1 dir_2 so I would expect the format to work just fine. Could you share these additional details and I can help troubleshoot why it's not working for you?

• Which operating system are you using?
• How did you format the command?
• What is the filepath that the log file resides in?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Pengor/chia-log-analysis/issues/17#issuecomment-852198085, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATY3AEDLIKN3KZTJLP53TS3TQTY57ANCNFSM453XJDTQ .

[Pengor]

Okay it would seem the issue is that the PowerShell plotter log file does not use the more common UTF-8 character encoding (which the other plotter logs do), I will look into adding support for 16-bit character encodings.

[denniskhk]

Hi, thank you for your effort in analysing this! Does that mean if I open and resave the log in UTF-8 it would work? I'll try that in a bit!

On Tue, 8 Jun 2021, 00:56 Drew M Johnson, @.***> wrote:

Okay it would seem the issue is that the PowerShell plotter log file does not use the more common UTF-8 character encoding (which the other plotter logs do), I will look into adding support for 16-bit character encodings.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Pengor/chia-log-analysis/issues/17#issuecomment-856103407, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATY3AEHMAC4YTHQIZPY6DFTTRT24PANCNFSM453XJDTQ .

[Pengor]

Yes, I was able to parse the log once it was converted to UTF-8. I think this is why I was unable to reproduce your issue originally, I probably copied and pasted the log file's contents (which likely converted it to UTF-8) rather than use the file itself.

I also noticed the line endings in your log file were CRLF (Windows style) whereas the GUI plotter produces LF line endings (Unix style) but the application was able to handle both just fine so that is a non-issue.

[denniskhk]

Hi,

It works! I converted my Powershell log files to UTF-8 using the attached Powershell script (renamed extension to .txt otherwise the attachment gets blocked) and I was able to parse the log files successfully. Now that you mention it I did take a look at the encoding and confirmed that the Powershell logs are indeed in UTF-16 (specifically, UCS-2 LE BOM apparently). This has been a most productive discussion as I wasn't keen on going back to manual tabulation of the log data - with 400+ files in waiting!

Out of curiosity, how did you figure out it was the encoding that was the issue?

Thank you and best regards, Dennis

On Tue, 8 Jun 2021 at 01:41, Drew M Johnson @.***> wrote:

Yes, I was able to parse the log once it was converted to UTF-8. I think this is why I was unable to reproduce your issue originally, I probably copied and pasted the log file's contents (which likely converted it to UTF-8) rather than use the file itself.

I also noticed the line endings in your log file were CRLF (Windows style) whereas the GUI plotter produces LF line endings (Unix style) but the application was able to handle both just fine so that is a non-issue.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Pengor/chia-log-analysis/issues/17#issuecomment-856134520, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATY3AECDHTKM4G3U2DGUJFTTRUAFZANCNFSM453XJDTQ .

Get-ChildItem "C:\path\" -Filter *.log -recurse | Foreach-Object { Write-Host (Get-Date).ToString() $.FullName Get-Content $.FullName | Set-Content -Encoding utf8 ($_.FullName + '_UTF8.log') }