Adding utf-8 as a supported string encoding when dumping

[sa3dany]

I am using this library to read and write to shortcuts.vdf which Steam uses to store non-Steam game shortcuts. While testing, I found that encoded non-ASCII strings were using utf-16 which caused Steam to incorrectly parse the file.

https://github.com/ValvePython/vdf/blob/f2fd3da0e214d6d3b54fc7d865ce1dc6494dd4d9/vdf/__init__.py#L368

I propose adding a new optional parameter to binary_dumps() (with utf-16 as a default) to allow encoding using utf-8. I have tested this in my fork of this library and Steam successfully parsed the resulting vdf file.

[rossengeorgiev]

Hello @sa3dany, you are saying that if I encoded a non-ascii named shortcut.vdf steam won't like it? Only wants WIDECHARS ?

I see that the dump func encodes in ascii, while the load decodes in utf-8. Have you tried simply changing the lines below from ascii to utf-8?

https://github.com/ValvePython/vdf/blob/f2fd3da0e214d6d3b54fc7d865ce1dc6494dd4d9/vdf/__init__.py#L379

https://github.com/ValvePython/vdf/blob/f2fd3da0e214d6d3b54fc7d865ce1dc6494dd4d9/vdf/__init__.py#L393

I expect that will work, although it will make the code for BIN_WIDESTRING unreachable.

[sa3dany]

Hi @rossengeorgiev, thank you for getting back to me so quickly. Decoding works correctly. The issue is when a string contains a non-ASCII character (for example, the ™ symbol) the value.encode('ascii') call will fail and then it will be encoded using utf-16 (which Steam parses incorrectly):

https://github.com/ValvePython/vdf/blob/f2fd3da0e214d6d3b54fc7d865ce1dc6494dd4d9/vdf/__init__.py#L392-L397

The change I did in my fork is that I added a new Boolean parameter called utf8 that allows encoding using utf-8 when ASCII fails.

Please let me if this makes sense. Thank you.