On Linux, command line arguments cannot contain multi-byte codepoints because of limitation on the String type

[lucabol]

This is the offending method, copilot suggests simple utf8 -> utf16 code, so maybe worth adding to libzero, but perhaps copilot is oversimplifying:

        private static unsafe string Ctor(sbyte* ptr)
        {
            sbyte* cur = ptr;
            while (*cur++ != 0) ;

            string result = FastNewString((int)(cur - ptr - 1));
            for (int i = 0; i < cur - ptr - 1; i++)
            {
                if (ptr[i] > 0x7F)
                    Environment.FailFast(null);
                Unsafe.Add(ref result._firstChar, i) = (char)ptr[i];
            }
            return result;
        }

[MichalStrehovsky]

You're running into the FailFast, right? Yeah, zerolib does cut corners like that. This constructor has different behaviors depending on whether this is Windows or Linux (it's "current codepage to UTF-16" on Windows, and "UTF-8 to UTF-16" on Linux).

[lucabol]

Well, it's not 'UTF-8 to UTF-16' on Linux, it is 'ASCII to UTF-16' (by design). It seems too limiting, even for zerolib. But perhaps you feel differently?

[MichalStrehovsky]

I'm not opposed to adding a helper for this. It might not be the only place where utf-8 to 16 would be useful.

[lucabol]

I am keeping track of everything that 'feels like' zerolib in a single file. Perhaps worth doing a simple PR, or discussion, when I am finished.

In case you wonder ... I am playing around with exposing utf8 cmd line args in the spirit of utf8everywhere, moving on experimenting with a no-allocation programming model (as MISRA C), to finish with a linear allocator. At least, this is the rough idea, we'll see.

[ghost]

On Windows, I think using the Windows API. On Linux, I think using libiconv. Don't write your own encoding converter.