Implement UTF-8 (without BOM) encoding support

[marwin89]

Suggestion

Introduce UTF-8 (Without BOM) Encoding for song textfiles as standard/default. Right now Vocaluxe only support UTF-8 (with BOM).

Why?

• UTF-8 (without BOM) is the only allowed encoding according to offical ultrastar song format specification. This is to end all encoding problems that heavily reduce user acceptance.
• CP encodings will be not possible in future in USDB.animux UTF-8 (without BOM) will be download standard in USDB and USDB SYNCER. All Song Editors agreed to UTF-8 (without BOM) as standard as well.
• All other games support UTF-8 (without BOM) Encoding
• This will highly improve acceptance of vocaluxe
• User can update collections from CP encoding to UTF-8 (Without BOM) via mass operation in UltraStar Manager.
• Developers don't have the problem to implement automatic guessing of encoding

Additional Info: vocaluxe is the only game that doesn't handle UTF-8 (Without BOM) at the moment

[marwin89]

@flokuep : could you estimate the effort of this issue on occassion maybe? (This confusion about encoding comes always up in help desk) 👋 😃

[flokuep]

Quick win: Move the prefered encoding for file loading into a configuration (and set UTF-8 as default if that's the USDB default) - Size: S

Alternative: Include a library that identify the file encoding so that all files could be loaded in the correct way. Check https://stackoverflow.com/questions/3825390/effective-way-to-find-any-files-encoding for some ideas. Size: M to L.

[Stefan1200-de]

Well, in fact Vocaluxe supports UTF-8 without BOM, but only if you add the following line to the txt file header:

ENCODING:UTF8

But of course, it would be better to switch to UTF8 as the default encoding. But this may break everything for current Vocaluxe users. So the best way would be to add a good working auto detection to Vocaluxe with UTF8 as default and fallback value.

Maybe I play around with that in the next week, but if someone already has a good idea, I would be open for a pull request too.