The old implementation (using grep .) had a weird edge case that caused half of my notes to be removed from the output JSON file. This was likely due to Unicode issues, as the note right after the cutoff had a metric ton of emojis and other unicode oddities.
The new version removes blank lines using sed, and also strips weird control characters from the end of each line as well using perl's negative-lookahead syntax. (this was likely a consequence of swapping grep out for sed)
With this change in place, the script should be more reliable, and should properly convert notes from the decrypted data into valid JSON without deleting half of them along the way.
The old implementation (using grep .) had a weird edge case that caused half of my notes to be removed from the output JSON file. This was likely due to Unicode issues, as the note right after the cutoff had a metric ton of emojis and other unicode oddities.
The new version removes blank lines using sed, and also strips weird control characters from the end of each line as well using perl's negative-lookahead syntax. (this was likely a consequence of swapping grep out for sed)
With this change in place, the script should be more reliable, and should properly convert notes from the decrypted data into valid JSON without deleting half of them along the way.