Unicode support for transaction imports

[shugyosha89]

Hi, thank you for your hard work creating OpenBudgeteer.

I am using it in Japan and would like to import transaction CSV data in Japanese, however the application currently does not support unicode. The text therefore becomes garbled before insertion into the database.

This issue likely affects many languages and locales.

Is it possible to change the expected encoding to UTF-8, or to allow the user to specify the encoding?

Thanks.

[TheAxelander]

Hi, thanks a lot for your feedback.

I'm wondering that there is an issue with UTF-8 as I thought it is working. Do you see the strange characters during import already in the Preview or does it only show up once the data have been finally loaded? Maybe it's just a collation setting of the used database and not the application itself.

[shugyosha89]

Hi, thanks for your reply. Yes, the strange characters appear already in the Preview, as well as after the data has been imported.

The database is using charset utf8mb4, so is capable of storing unicode characters.

[TheAxelander]

Ok good to know. May you can share any kind of sample file (of course with anonymized/random data), so I can test this on my side.

[TheAxelander]

Found the issue in OpenBudgeteer.Core.ViewModels.ImportDataViewModel.cs line 209,210,245

#209
FileText = File.ReadAllText(FilePath, Encoding.GetEncoding(1252));
#210
_fileLines = File.ReadAllLines(FilePath, Encoding.GetEncoding(1252));
#245
using var lineReader = new StreamReader(stream, Encoding.GetEncoding(1252));

Currently it reads the data as ISO 8859-1, would need to change and test it to UTF-8

[shugyosha89]

Thank you!

[TheAxelander]

A (hopefully) working Fix is now available in pre-release. Update 1.7 is planned to be released in a few days.