QNanoPainter cannot display unicode text

NielsMayer commented 1 year ago

When QNanoPainter text contains unicode, it is displayed as a box with an X through it.

Whereas a Text item in QML displays the Unicode correctly:

FYI, Original source of this text is from the title of this Japanese QtCo video: https://www.youtube.com/watch?v=UKEXot9z06I

NielsMayer commented 1 year ago

Normally I end up having the app translate to English so the lack of Unicode isn't a problem even if the video is in Japanese. However a native Japanese or Chinese user will not be happy seeing those boxed X's instead of their native language.

mariuszmaximus commented 1 year ago

I confirm the problem, this is probably a font problem If I have in one string Polish, Russian and Japanese , Polish and Russian is correct

mariuszmaximus commented 1 year ago

Default font is Roboto, but Roboto dont' have Japanese

void QNanoFont::setFontId(FontId fontId)
{
    switch (fontId)
    {
    case QNanoFont::DEFAULT_FONT_THIN:
        setFilename(":/qnanopainter/data/Roboto-Thin.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_THIN_ITALIC:
        setFilename(":/qnanopainter/data/Roboto-ThinItalic.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_LIGHT:
        setFilename(":/qnanopainter/data/Roboto-Light.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_LIGHT_ITALIC:
        setFilename(":/qnanopainter/data/Roboto-LightItalic.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_NORMAL:
        setFilename(":/qnanopainter/data/Roboto-Regular.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_NORMAL_ITALIC:
        setFilename(":/qnanopainter/data/Roboto-Italic.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_BOLD:
        setFilename(":/qnanopainter/data/Roboto-Bold.ttf");
        break;
    case QNanoFont::DEFAULT_FONT_BOLD_ITALIC:
        setFilename(":/qnanopainter/data/Roboto-BoldItalic.ttf");
        break;
    }
}

But is possible use any font for example https://www.1001fonts.com/jaycons-font.html

    QNanoFont font1("d:\\Jaycons.ttf");
    font1.setPixelSize(size*0.08);
    p->setFont(font1);
    p->setFillStyle("#B0D040");
    p->fillText("HELLO日本語の", center.x(), center.y() - size*0.18);

NielsMayer commented 1 year ago

So it probably makes sense to "close" this bug or change the title to "qnanopainter doesn't use internationalized fallback fonts for unicode that results in 'tofu', unlike chromium, openoffice, and other multilingual apps." And it might well be out of scope for QNanoPainter or rather NanoVG, to actually do this.

The included Roboto font is certainly sufficient for running QNanoPainter examples. Any other fonts with better language and/or symbol coverage are significantly larger. For example the font I selected to resolve this issue, NotoSansCJK-Regular.ttc is 19Mb, and that's compressed.

Links for the suggested "fallback" mechanism as well as other notes on the subject are enclosed in my own file fonts/README which I reproduce below:

"NotoSansCJK-Regular" was the only font of several "noto" fonts that at least displayed symbols like "♪", plus Japanese and Chinese text commonly output by whisper.cpp and whisper.

"noto" supposedly is multilingual -- the trick is to find the variant with good coverage of language commonly seen in subtitles and output by AI tools like Whisper.

https://en.wikipedia.org/wiki/Noto_fonts https://fonts.google.com/knowledge/using_type/language_support_in_fonts https://www.dezeen.com/2016/10/06/google-monotype-noto-font-biggest-typeface-all-written-language/# https://designmodo.com/noto-font/ https://fonts.google.com/knowledge/choosing_type/choosing_reliable_typefaces

For example https://community.wanikani.com/t/googles-noto-fonts-are-really-beautiful/19295 says

“Noto Sans CJK and Noto Serif CJK comprehensively cover Simplified Chinese, Traditional Chinese, Japanese, and Korean in a unified font family. This includes the full coverage of CJK Ideographs with variation support for 4 regions, Kangxi radicals, Japanese Kana, Korean Hangul, and other CJK symbols and letters in the Basic Multilingual Plane of Unicode. It also provides limited coverage of CJK Ideographs in Plane 2 of Unicode as necessary to support standards from China and Japan.” These fonts, in regards to Japanese: “Supports all of the kanji in JIS X 0208, JIS X 0213, and JIS X 0212 to include all kanji in Adobe-Japan1-6.”

Note that the better symbol coverage of /usr/share/fonts/truetype/freefont/FreeSans.ttf from ubuntu deb "fonts-freefont-ttf" and better asian language coverage by /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc from ubuntu deb package 'fonts-noto-cjk' is explained by size, versus other fonts I tested:

  -rw-r--r--  1 root root 837952   Oct 23 23:17 FreeSans.ttf
  -rw-r--r--  1 root root 356760   Oct 23 22:29 DejaVuSans-ExtraLight.ttf
  -rw-r--r--  1 root root 107848   Oct 23 22:05 NotoMono-Regular.ttf
  -rw-rw-r--  1 root root 556216   Oct  5 14:39 NotoSans-Regular.ttf
  -rw-rw-r--  1 root root 409140   Oct  5 14:39 NotoSans-ThinItalic.ttf
  -rw-r--r--  1 root root 19484784 Jan 27  2022 /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc

//      Original code using default font displays "Tofu" for Chinese, Japanse or Symbols generated by whisper transcription.
//      QNanoFont titleFont(QNanoFont::DEFAULT_FONT_THIN_ITALIC);
//      QNanoFont titleFont = QNanoFont(":/trainspodder/fonts/NotoSans-ThinItalic.ttf");
//      QNanoFont titleFont = QNanoFont(":/trainspodder/fonts/NotoSans-Regular.ttf");
//      QNanoFont titleFont = QNanoFont(":/trainspodder/fonts/NotoMono-Regular.ttf");
//      QNanoFont titleFont = QNanoFont(":/trainspodder/fonts/DejaVuSans-ExtraLight.ttf");
    // all the above "Noto" fonts don't display Japanese or the "♪" symbol output by whisper.cpp during music
    // However, symbols are displayed in the caption font used by VLC
    // /usr/share/vlc/skins2/fonts: FreeSans.ttf -> ../../../fonts/truetype/freefont/FreeSans.ttf
    // which points to /usr/share/fonts/truetype/freefont/FreeSans.ttf from ubuntu deb "fonts-freefont-ttf"
//      QNanoFont titleFont = QNanoFont(":/trainspodder/fonts/FreeSans.ttf");
    // However, even better direct coverage is found for Asian languages
    // and glyphs like "♪" in deb package 'fonts-noto-cjk' 
        // file /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
        QNanoFont titleFont = QNanoFont(":/trainspodder/fonts/NotoSansCJK-Regular.ttc");

alas "FreeSans" still doesn't display Japanese (e.g. https://www.qt.io/ja-jp/blog/qt%E3%81%AE%E9%9D%99%E7%9A%84%E8%A7%A3%E6%9E%90%E3%83%84%E3%83%BC%E3%83%ABaxivion-suite%E5%85%A5%E9%96%80-%E3%81%9D%E3%81%AE%EF%BC%91 )
multilinguality is done via fallback to other fonts within the given font class that have a non-tofu representation of the missing char or glyph.
in apps like libreoffice, chromium, firefox, a "fallback mechanism" is employed where if the given font produces "tofu" other types of ttf's related to the font are searched:

https://github.com/podlove/podlove-subscribe-button/issues/110 https://ask.libreoffice.org/t/noto-fonts-regular-are-not-displaying-some-characters-in-libre-office-squares-instead/61485/11 https://stackoverflow.com/questions/56847665/how-do-browsers-deal-with-tofu-characters

QUItCoding / qnanopainter

QNanoPainter cannot display unicode text #74