Crashing with Diacritics

[JranZu]

It took me forever to track this down, but submitting a prompt with a diacritic (é, ç, etc.), it throws a corrupted memory exception. I was able to fix this by filtering my prompts when they come in. Just wanted to let you know:

private static string ReplaceDiacritics(string text)
{
    if (string.IsNullOrWhiteSpace(text))
    {
        return text;
    }

    // Normalize the text to decompose accents into separate characters
    var normalizedString = text.Normalize(NormalizationForm.FormD);
    int length = normalizedString.Length;

    // Use string.Create for optimized memory allocation
    return string.Create(length, normalizedString, (span, value) =>
    {
        int index = 0;
        foreach (var c in value)
        {
            // Directly check for non-spacing marks and only copy valid characters
            if (CharUnicodeInfo.GetUnicodeCategory(c) != UnicodeCategory.NonSpacingMark)
            {
                span[index++] = c;
            }
        }

        // Truncate the span to the valid length and normalize back to Form C
        var cleanedSpan = span.Slice(0, index);
        var cleanedString = new string(cleanedSpan);
        cleanedString.Normalize(NormalizationForm.FormC).CopyTo(span);
    });
}

[DarthAffe]

This is weird. It seems to be either model or backend dependant. I tried different things, but for me these characters work fine (like it does not crash, they don't produce good results).

[JranZu]

Weird - mine crash everytime on every model I have tried; using Cuda12 Windows. If you can't repro it feel free to close this.