Automatic Unicode detection is messing up some files

[sebcworks]

I had some problems with simple ASCII text files that were interpreted as Unicode files.

For example, a text file like this one:

#> file 20150201000000.datebound.temp
20150201000000.datebound.temp: ASCII text

#> od -bc 20150201000000.datebound.temp | head
0000000 062 060 061 065 055 060 062 055 060 061 040 060 060 072 060 071
          2   0   1   5   -   0   2   -   0   1       0   0   :   0   9
0000020 011 060 061 057 060 062 057 062 060 061 065 040 060 060 072 060
         \t   0   1   /   0   2   /   2   0   1   5       0   0   :   0
0000040 071 072 060 060 011 060 054 060 011 060 054 060 011 060 054 060
          9   :   0   0  \t   0   ,   0  \t   0   ,   0  \t   0   ,   0
0000060 011 060 054 060 011 060 054 060 011 060 054 060 011 060 054 060
         \t   0   ,   0  \t   0   ,   0  \t   0   ,   0  \t   0   ,   0

#>  od -bc 20150201000000.datebound.temp | tail
7705240 063 072 065 071 011 063 060 057 060 064 057 062 060 061 065 040
          3   :   5   9  \t   3   0   /   0   4   /   2   0   1   5
7705260 062 063 072 065 071 072 060 060 011 060 054 060 011 060 054 060
          2   3   :   5   9   :   0   0  \t   0   ,   0  \t   0   ,   0
7705300 011 060 054 060 011 060 054 060 011 060 054 060 011 060 054 060
         \t   0   ,   0  \t   0   ,   0  \t   0   ,   0  \t   0   ,   0
*
7705360 011 060 054 070 066 070 060 065 063 066 067 064 066 071 070 012
         \t   0   ,   8   6   8   0   5   3   6   7   4   6   9   8  \n
7705400

That should be displayed like that (and that is displayed correctly if I disable automatic Unicode detection):

Is in fact displayed like that with automatic Unicode detection:

If you need any more details on the files (the one I gave is not the only one), feel free to ask...

Anyway, thanks for this fork, it is amazing :)

[XhmikosR]

You can check if this happens with vanilla Notepad2 and Scite.

[sebcworks]

Scite is perfectly fine but vanilla Notepad2 does not open the file at all (not responding -> crash).

btw, I forgot to tell, but I'm using x64 version (the latest in both case, vanilla 4.2.25 and 4.2.25 r964 for the mod), Scite 3.6.2 standalone exe.