Unicode BOM characters randomly inserted into ANSI Virtual Node text files

[GoogleCodeExporter]

- KeyNote NF v1.7.8.1 on Win7 64-bit (normal NTFS partitions)

Since switching to KeyNote NF (great work Daniel, thank you!) I'm seeing random 
fairly frequent weird corruption in the text of some nodes, which I've 
established is being caused by Unicode BOM characters being inserted at random 
points in ANSI Virtual Node text files.

Here's the hopefully reproducible steps:

1. Create new blank text document in Notepad.
2. Link to as Virtual Node in KeyNote NF.
3. Go to this SmashingMagazine.com article (random page I found that triggers 
this): http://bit.ly/DNyW2
4. Press CTRL+A then CTRL+C to copy the page contents.
5. Switch to the new Virtual Node in KeyNote, paste the clipboard contents.
6. Text will appear fine for the moment - but switch nodes (so saves) and open 
the original text file. You'll see the weird text size corruption on "This is 
an alternative to the Posterization m?ethod .." and it will claim UTF-8 format 
when you Save As. 

If you look at the same line in KeyNote, you'll notice character corruption on 
the word "method". I've attached the tex file as an example ("test3.txt") - if 
you look in a hex editor, you'll notice [EF BB BF] has been added after the "e" 
in "method" on the corrupted line. The same occurs on the other corrupted lines 
(often repeated multiple times).

Note that random corruption like the above doesn't just appear to affect pasted 
text. I regularly type out notes by hand, then return to the section a few days 
later to see character corruption (which I now presume is caused by randomly 
inserted BOM marks).

This is clearly a bug - hopefully the above is enough to identify a fix? Many 
thanks for your great work Daniel!

Original issue reported on code.google.com by psd...@gmail.com on 29 Aug 2011 at 10:39

Attachments:

• test3.txt

[GoogleCodeExporter]

Further discussion in forum, including poss temporary workarounds:
http://sourceforge.net/apps/phpbb/keynote-newfeat/viewtopic.php?f=15&t=50

Original comment by psd...@gmail.com on 30 Aug 2011 at 12:08

[GoogleCodeExporter]

Hello
I will study it. Anyway, the use of UTF8 BOM is not necessary an error. As a 
consecuence of Issue #139 (Make KeyNote NF Unicode compliant) I made some 
little adjustments in KNT file format.

Please, see the comments in that issue. Also, inside the file name 
'fileformat_NEW.txt' I included the following comment:

<<From version 1.7.7, certain metainformation is encoded in UTF-8 without BOM, 
not in ASCII.
They are the corresponding to the following keys:
NN, ND, VN, RV, VF, NA, EN
>>

Regards
Daniel

Original comment by dpra...@gmail.com on 31 Aug 2011 at 4:07

• Changed state: Evaluating

[GoogleCodeExporter]

Hi Daniel - just see your comment thanks. To clarify, the UTF8 BOM is not being 
pasted in or anything like that - even in the pasting example above, there's no 
BOM mark in the copied content. Instead some bug is causing the BOM to be 
randomly inserted throughout various nodes (+ notes?) without even editing 
those nodes. I'd be *very* surprised if this is expected behaviour.

Fingers crossed for tracking down a fix this year. Many thanks,  Ben

Original comment by psd...@gmail.com on 4 Feb 2012 at 1:24

[GoogleCodeExporter]

Hello Ben, I misunderstood your problem. Now I understand it and will be verify 
and correct it if necessary

Regards

Original comment by dpra...@gmail.com on 7 Jan 2013 at 1:12

• Changed state: Accepted
• Added labels: Milestone-1.7.9

[GoogleCodeExporter]

Fixed in revision r172
In this example the text will be saved as UTF8 because includes unicode 
characters (paste it directly into notepad), but now will only insert UTF8 BOM 
at the beginnig of the file. As you indicated, it was inserting erroneously 
many UTF8 BOM characters. 

Original comment by dpra...@gmail.com on 7 Jan 2013 at 5:02

• Changed state: Finished

[GoogleCodeExporter]

I have just released a new version of KeyNote NF: 1.7.9 Beta 4

Could you verify and confirm if the issues are correctly resolved?
Thanks
Daniel

Original comment by dpra...@gmail.com on 7 Jan 2013 at 7:43