UTF-8 and UTF-16 native string support for performance

zeromq / czmq

High-level C binding for ØMQ

czmq.zeromq.org

Mozilla Public License 2.0

1.16k stars 523 forks source link

UTF-8 and UTF-16 native string support for performance #2194

Open xbotuk opened 2 years ago

xbotuk commented 2 years ago

As sometimes the source and target systems could be using native UTF-16 strings. In some instances one is UTF-8 and other is UTF-16.

Can you please add support to configure native string supports or ways to minimise the unnecessarily string conversions overheads?

sphaero commented 2 years ago

Any suggestions how to implement this. PR's are welcomed

sphaero commented 2 years ago

Perhaps as a reference:

"In particular, we believe that the very popular UTF-16 encoding (often mistakenly referred to as ‘widechar’ or simply ‘Unicode’ in the Windows world) has no place in library APIs except for specialized text processing libraries, e.g. ICU."

from http://utf8everywhere.org/

Which argues to use char* with utf-8 everywhere.