Tamil font not displayed correctly

[muruga86]

I am writing a small program in python using PIL library to render a Tamil text on an image using couple of Tamil fonts. However, certain letters in Tamil are not displayed properly. Some characters are not correctly or aesthetically printed on image like பி, ழ், த், து, யோ, கு, டி. Need to use any reshaper or any other thing in code. Could some one throw light on this issue? Even tried layout_engine=ImageFont.LAYOUT_RAQM, but still same issue

Example: இனிய பிறந்தநாள் வாழ்த்துக்கள் யோஷினி குட்டி

Displayed as:

Correct display:

Code:

import numpy as np
import PIL 
from PIL import ImageFont, ImageDraw, Image
import cv2

## Make canvas and set the color
img = np.zeros((720, 1280,3),np.uint8)
b,g,r,a = 0,0,0,255 #black

#fontpath = "C:/Ubuntusharefolder/CustomFonts/akshar.ttf" # <== தமிழ்
#fontpath = "C:/Ubuntusharefolder/CustomFonts/Coiny-Regular.ttf" # <== தமிழ்
#fontpath = "C:/Ubuntusharefolder/CustomFonts/NotoSansTamil-Regular.ttf" # <== தமிழ்
fontpath = "C:/Ubuntusharefolder/CustomFonts/latha/latha.ttf" # <== தமிழ்

print(">>>font path:", fontpath)
#font = ImageFont.truetype(fontpath, 32)
font = ImageFont.truetype(fontpath, 32, layout_engine=ImageFont.LAYOUT_RAQM) 
#Tamil fonts use advanced true type features for rendering. LAYOUT_RAQM should do the trick
#font = ImageFont.truetype(fontpath, 32, encoding='unic',layout_engine=ImageFont.LAYOUT_RAQM) 

#line = "இனிய பிறந்தநாள் வாழ்த்துக்கள் யோஷினி குட்டி"
line1 = "இனிய பிறந்தநாள் வாழ்த்துக்கள்"
line2 = "யோஷினி குட்டி"

line2.encode('raw-unicode-escape').decode('utf-8')
img_pil = Image.fromarray(img)
draw = ImageDraw.Draw(img_pil)
b,g,r,a = 255,255,255,0 #white
y_pos = 200
x_pos = 100

draw.text((x_pos, y_pos),  line1, font = font, fill = ((b, g, r, a)))
draw.text((x_pos, y_pos+100),  line2, font = font, fill = ((b, g, r, a)))
res_img = np.array(img_pil)

## Display 
cv2.imshow("result", res_img);cv2.waitKey();cv2.destroyAllWindows()
#cv2.imwrite("result.png", res_img)]

[muruga86]

@radarhere Can you point me the code where the changes should probably get in, so that the fix can be done from my end, validated and pushed to repo.

[nulano]

Are you asking where text rendering is implemented in Pillow? src/PIL/ImageDraw.py calls methods of font objects which are defined in src/PIL/ImageFont.py, but most of the text rendering code is in src/_imagingft.c. Text shaping is handled via Raqm by HarfBuzz.

[muruga86]

I've cloned the latest pillow github repo using github desktop and used Anaconda3 with vscode editor (Windows10) for compiling the repo. Also cloned zlib and libjpeg repo and linked those two mandatory include/libraries as well, tweaked pillow setup.py to include those 2 libraries and compiled the pillow binary successfully.

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.1.0.dev0
platform     win32 3.8.8 (default, Apr 13 2021, 15:08:03)
             [MSC v.1916 64 bit (AMD64)]
--------------------------------------------------------------------
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
*** LIBTIFF support not available
*** FREETYPE2 support not available
*** RAQM (Text shaping) support not available
*** LITTLECMS2 support not available
*** WEBP support not available
*** WEBPMUX support not available
*** XCB (X protocol) support not available
--------------------------------------------------------------------

sample import code:

sys.path.insert(1,'C:/Users/Administrator/Documents/GitHub/Pillow/build/lib.win-amd64-3.8')
from PIL import ImageFont, ImageDraw, Image

However, after importing the compiled library in to a sample program and while triggering execution, get an error like

PS C:\Users\Administrator\Documents\GitHub\Pillow> & python c:/Users/Administrator/Documents/GitHub/tamilfontexp.py
>>>font path: C:/Ubuntusharefolder/CustomFonts/arial-unicode-ms.ttf
Traceback (most recent call last):
  File "c:/Users/Administrator/Documents/GitHub/tamilfontexp.py", line 25, in <module>
  File "C:/Users/Administrator/Documents/GitHub/Pillow/build/lib.win-amd64-3.8\PIL\ImageFont.py", line 841, in freetype
    return FreeTypeFont(font, size, index, encoding, layout_engine)
  File "C:/Users/Administrator/Documents/GitHub/Pillow/build/lib.win-amd64-3.8\PIL\ImageFont.py", line 169, in __init__
    if core.HAVE_RAQM:
  File "C:/Users/Administrator/Documents/GitHub/Pillow/build/lib.win-amd64-3.8\PIL\ImageFont.py", line 43, in __getattr__
    raise ImportError("The _imagingft C module is not installed")
ImportError: The _imagingft C module is not installed

On further checking, get suggestions like libfreetype6 should be preinstalled before compilation of PIL (not sure whether this package is mandatary). Could any one suggest me some thing to solve this issue?

[nulano]

The full installation instructions are here: https://pillow.readthedocs.io/en/stable/installation.html Windows specific build instructions are here: https://github.com/python-pillow/Pillow/blob/main/winbuild/build.rst

Edit: In particular, FreeType is required for text rendering, and HarfBuzz, FriBiDi and Raqm are required for Raqm text shaping (there is a copy of raqm sources in Pillow sources that can be used instead if you pass the correct build flag - should be automatic if you use winbuild/build_prepare as described above). libfreetype6 is the name of the Linux build of FreeType, on Windows you need freetype.lib or freetype.dll (again, winbuild/build_prepare takes care of that for you).

[nulano]

When I try this, I get an image that looks like your "correct" image. Am I missing something, or is this what you would expect?

from PIL import Image, ImageDraw, ImageFont

text = "இனிய பிறந்தநாள் வாழ்த்துக்கள் யோஷினி குட்டி"
font = ImageFont.truetype(r"D:\Downloads\latha.ttf", 32, layout_engine=ImageFont.LAYOUT_RAQM)

im = Image.new("RGB", (1280, 720), "black")
draw = ImageDraw.Draw(im)
draw.text((100, 200), text, font=font, fill="white")
im.show()

I only get the "incorrect" image if I specify font = ImageFont.truetype(..., layout_engine=ImageFont.LAYOUT_BASIC):

Perhaps your problem is just that you are missing FriBiDi? However, the code you provided should raise an exception if you are missing FriBiDi (when creating the font). Edit: Checking again I see that ImageFont falls back to basic layout silently if Raqm is missing.

You can obtain fribidi.dll using the scripts in winbuild, or otherwise, and place it in a directory listed in the Dynamic-Link Library Search Order (Microsoft Docs and restart Python to enable Raqm support if you are using the provided Pillow wheels from PyPI.

[muruga86]

@nulano Thanks for the inputs, yes the first image is the expected one. However, I am unable to get that expected image in my setup.

I tried the steps mentioned in https://github.com/python-pillow/Pillow/blob/main/winbuild/build.rst link, able to execute

1. GitHub/Pillow/winbuild/build_prepare - OK
2. winbuild\build\build_dep_all.cmd - OK
3. winbuild\build\build_pillow.cmd install - FAIL (error LNK2001: unresolved external symbol)
4. winbuild\build\build_pillow.cmd --inplace develop - FAIL (error LNK2001: unresolved external symbol)

Error:

Creating library build\temp.win-amd64-3.8\Release\src\_imaging.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\src\_imaging.cp38-win_amd64.exp
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_alloc_compressor
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_alloc_compressor
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_zlib_compress
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_zlib_compress
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_zlib_compress_bound
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_zlib_compress_bound
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_free_compressor
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_free_compressor
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_alloc_decompressor
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_alloc_decompressor
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_zlib_decompress
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_zlib_decompress
tiff.lib(tif_zip.c.obj) : error LNK2001: unresolved external symbol libdeflate_free_decompressor
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol libdeflate_free_decompressor
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol lerc_encodeForVersion
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol lerc_getBlobInfo
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol lerc_decode
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol ZSTD_compress
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol ZSTD_decompress
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol ZSTD_compressBound
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol ZSTD_isError
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_isError
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol ZSTD_getErrorName
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_getErrorName
tiff.lib(tif_lerc.c.obj) : error LNK2001: unresolved external symbol ZSTD_maxCLevel
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_maxCLevel
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_createCStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_freeCStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_initCStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_compressStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_endStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_createDStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_freeDStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_initDStream
tiff.lib(tif_zstd.c.obj) : error LNK2001: unresolved external symbol ZSTD_decompressStream
build\lib.win-amd64-3.8\PIL\_imaging.cp38-win_amd64.pyd : fatal error LNK1120: 25 unresolved externals
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120

To remove confusion, I tried to execute the same code in google colab and get the same issue originally posted https://colab.research.google.com/drive/17X1eoXjLc1g0j_uREdmMKu_7_2Q-mlyd?usp=sharing Output image:

Not sure where the problem lies...

[nulano]

Not sure why tiff is failing to link, but you could just disable it by running winbuild\build\build_pillow.cmd --disable-tiff install.

---

Looking at Colab (which I remember doesn't support Raqm, at least not by default), it seems I was wrong about the behaviour when Raqm is not available. You have to check yourself whether it was selected, for example by adding assert font.layout_engine == ImageFont.LAYOUT_RAQM to the script.

Since winbuild\build\build_dep_all.cmd worked for you, you should now have a file winbuild/build/bin/fribidi.dll that you can copy to a directory in the Dynamic-Link Library Search Order (Microsoft Docs) (I usually use C:\Program Files\PythonXY\fribidi.dll). That should be enough to get Raqm support using the wheel from PyPI.

[muruga86]

@nulano

Thanks for your help, build_pillow.cmd --disable-tiff and fribidi.dl copy to dll folder steps solved both the compilation issue and able to execute and get expected output (after importing PIL from generated build).

With generated PIL (expected): sys.path.insert(1,'C:/Users/Administrator/Documents/GitHub/Pillow/build/lib.win-amd64-3.8')

With default PIL (incorrect not expected):

Regarding the colab part, I have also added assert line for not equal LAYOUT_RAQM, but it failed. This confirms in colab LAYOUT_RAQM function works properly, but still rendering is not as per expectation. assert font.layout_engine != ImageFont.LAYOUT_RAQM, "Assert ImageFont.LAYOUT_RAQM failure"

My only concern is the feature is working well only in development build, but not in released build - both in windows/linux (colab). Hence, need to investigate further to figure out the cause of the issue.

[nulano]

Regarding the colab part, I have also added assert line for not equal LAYOUT_RAQM, but it failed. This confirms in colab LAYOUT_RAQM function works properly, but still rendering is not as per expectation. assert font.layout_engine != ImageFont.LAYOUT_RAQM, "Assert ImageFont.LAYOUT_RAQM failure"

Your assert is wrong. As written, it will only raise if Raqm is NOT supported. The correct assert would be

assert font.layout_engine == ImageFont.LAYOUT_RAQM, "Assert ImageFont.LAYOUT_RAQM failure"

which is equivalent to

if font.layout_engine != ImageFont.LAYOUT_RAQM:
    raise AssertionError("Assert ImageFont.LAYOUT_RAQM failure")

Alternatively, you can check for Raqm before you create a font. From https://pillow.readthedocs.io/en/stable/reference/ImageFont.html#PIL.ImageFont.truetype:

You can check support for Raqm layout using PIL.features.check_feature() with feature="raqm".

import PIL.features
assert PIL.features.check_feature("raqm"), "Raqm support is missing"

or

import PIL.features
if not PIL.features.check_feature("raqm"):
    raise Exception("Raqm support is missing")

---

With default PIL (incorrect not expected):

My only concern is the feature is working well only in development build, but not in released build - both in windows/linux (colab). Hence, need to investigate further to figure out the cause of the issue.

My picture above was created with the official 9.0.1 wheel on Windows. Remember you need fribidi.dll for all Windows builds by default, even with the official wheel.

On Linux you have two options:

• if using an official binary wheel (version 8.2.0 or newer) you just need to install fribidi on your system
• if using a source build, you need to install all of harfbuzz, fribidi, and raqm, then reinstall Pillow

Raqm support on Google Colab is tricky as they are preinstalling a very old version of Pillow (currently 7.1.2). This version requires installing Raqm, but then crashes when it is used (this was fixed in Pillow 8.2.0): https://colab.research.google.com/drive/18Wx98Zk7kBBSlWJ3u362SJ1sxAO4aant?usp=sharing

Instead, if you wish to use Raqm on Google Colab, you have to update Pillow. However, as Pillow is being imported by default, this will cause strange crashes unless you also restart the runtime. After restarting, you can finally get the correct output: https://colab.research.google.com/drive/1y7HGr_eqGEekDtxnnkXJMCt4NjYmSHVI?usp=sharing

[nulano]

Perhaps your problem is just that you are missing FriBiDi? However, the code you provided should raise an exception if you are missing FriBiDi (when creating the font). Edit: Checking again I see that ImageFont falls back to basic layout silently if Raqm is missing.

I have created #6035 to add a warning in this case.

[muruga86]

I think you may be right about colab part.

Regarding the fribidi.dll, as mentioned it has been copied to default windows dll folder, hence should be available to all applications. However, the expected behavior is obtained only when the PIL import path is given as newly generated library instead of default one. My only concern is in both the cases (PIL from generated path vs PIL from default path) both have access to fribidi.dll for use, so there is something else we might be overlooking...

[nulano]

What do you get when you run python -m PIL? (only the top part is relevant here, supported file formats are not important)

If you have the latest wheel, it should say that you have Pillow 9.0.1. If that does not detect Raqm, there might be a hint for why in the used paths listed there. As stated above, the official wheel is working for me (I used Python 3.7 x64. I see you have Python 3.8 x64, but that should not make a difference unless they were built differently for some reason).

[muruga86]

PS C:\Users\Administrator\Documents\GitHub\Pillow> python -m PIL
--------------------------------------------------------------------
Pillow 8.2.0
Python 3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)]
--------------------------------------------------------------------
Python modules loaded from C:\ProgramData\Anaconda3\lib\site-packages\PIL
Binary modules loaded from C:\ProgramData\Anaconda3\lib\site-packages\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 8.2.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.10.4
*** LITTLECMS2 support not installed
*** WEBP support not installed
*** WEBP Transparency support not installed
*** WEBPMUX support not installed
*** WEBP Animation support not installed
--- JPEG support ok, compiled for 9.0
*** OPENJPEG (JPEG2000) support not installed
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.11
--- LIBTIFF support ok, loaded 4.2.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
*** XCB (X protocol) support not installed
--------------------------------------------------------------------

[nulano]

Python modules loaded from C:\ProgramData\Anaconda3\lib\site-packages\PIL

Did you install Pillow using Anaconda? I am not too familiar with Anaconda, but it wouldn't surprise me to learn that they build Pillow without Raqm support (and many other formats it seems from the above).

I expect you should have no problem with Raqm if you instead install Pillow using a wheel from PyPI using pip.

[muruga86]

I think the latest version supported by pip is 8.2.0.

PS C:\Users\Administrator\Documents\GitHub\Pillow> pip install Pillow
Requirement already satisfied: Pillow in c:\programdata\anaconda3\lib\site-packages (8.2.0)
PS C:\Users\Administrator\Documents\GitHub\Pillow> 

PS C:\Users\Administrator\Documents\GitHub\Pillow> conda install -c conda-forge Pillow
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

[nulano]

Pip does not attempt to upgrade a package by default, you have to use --upgrade or -U parameter:

python -m pip install -U Pillow

It might be a good idea to uninstall Pillow using conda first.

You could try upgrading Pillow using conda (conda-forge should have 9.0.1), but I don't know the required command, and it probably won't have Raqm support anyway.

[radarhere]

@muruga86 it sounds to me like the latest Pillow is working correctly. Do you agree?

If you have any issues specifically with the Anaconda distribution of Pillow, https://github.com/conda-forge/pillow-feedstock is the place to talk about them.

[muruga86]

Yes, I too think so, we can close this ticket.