Problem with dIsSpace with UTF8 text

[jamesu]

If you attempt to use dIsspace on UTF8 text which uses high ranged unicode characters (e.g. japanese text), you may get an assert in a DEBUG build on windows due to the character value being out of range for ASCII. This prevents such text from being passed to several functions making it slightly impractical to use international text in Torque3D, at least in windows.

I should also note that technically calling dIsspace on a UTF8 character is not quite correct as there are several space characters in unicode which don't fit into a single UTF8 character.

Affected functions:

• TrimFirstWord
• CodeBlock::exec (OP_ITER)
• isspace ConsoleFunction
• sParseList
• sGetArgNameAndType
• Namespace::Entry::getDocString
• parseFunctionArgumentNames
• GFont::wrapString
• GFXGLShader::_handleIncludes
• GuiTextEditCtrl::findPrevWord
• String::trim
• String::GetTrailingNumber
• nat_isspace
• SplitSequencePathAndName

[crabmusket]

So this requires replacing the use of dIsSpace with an equivalent that will work for UTF8 strings?

[jamesu]

Possible solutions:

• Use UTF16 text internally (probably not practical at this stage)
• Use a simpler check which just checks for tabs, whitespace, etc

Nothing else is really makes much sense as dIsSpace only accepts char. Likely the intent of the code is only to accept basic spaces anyway.