Outline.decompose() fails for multiprocessing

[rardz]

Outline.decompose() fails with Multiprocessing module, here is the a piece of code modified from the official example, runs with Multiprocessing.Pool. As long as long I comment the decompse line in font_process, all things got OK, while run with it processes will got blocked permanently.

import os
import freetype as ft
import multiprocessing

def move_to(a, ctx):
    ctx.append("M {},{}".format(a.x, a.y))

def line_to(a, ctx):
    ctx.append("L {},{}".format(a.x, a.y))

def conic_to(a, b, ctx):
    ctx.append("Q {},{} {},{}".format(a.x, a.y, b.x, b.y))

def cubic_to(a, b, c, ctx):
    ctx.append("C {},{} {},{} {},{}".format(a.x, a.y, b.x, b.y, c.x, c.y))

def font_process(ft_path):
    ft_face = ft.Face(ft_path)
    ft_face.set_char_size( 18*64 )

    ft_face.load_char('B', ft.FT_LOAD_DEFAULT | ft.FT_LOAD_NO_BITMAP)
    ctx = []
    ft_face.glyph.outline.decompose(ctx, move_to=move_to, line_to=line_to, conic_to=conic_to, cubic_to=cubic_to)
    print(f'{os.path.basename(ft_path)} is finished.')

if __name__ == '__main__':
    num_workers = 4
    ft_path_list = ['./a.ttf', './b.ttf', './c.ttf', './d.ttf']

    p = multiprocessing.Pool(num_workers)
    for ft_path in ft_path_list:
        p.apply_async(font_process, args=(ft_path,))
    p.close()
    p.join()

[HinTak]

I think this is a known limitation of freetype/freetype-py. For such usage, you should use a new library handle in each thread - it is a hidden _lib variable in the python code, but you can get a new one yourself.

[HinTak]

You need to assign a new lib, I think, in each thread: https://github.com/rougier/freetype-py/blob/c0814fcf6feeda4ab453feb57274715989b6e3c2/freetype/raw.py#L49

[rardz]

You need to assign a new lib, I think, in each thread:

https://github.com/rougier/freetype-py/blob/c0814fcf6feeda4ab453feb57274715989b6e3c2/freetype/raw.py#L49

Thanks very much. However I still get no point to how to assigning such a hidden _lib in python code, can you please give more details.

Even it was the right way to mitigate the problem, I still have no idea why other thing like load_char works fine for mutiprocessing? And why the assignment work haven't be done by every new Face instantiation, since from source it seems like Face wrapper manage such low level things.

[HinTak]

I think if you move the import freetype as ft to inside the font_processing routine, that might cause a new handle to be assigned per thread, and fix your issue.

Or do something more dramatic like ft._lib = ctypes.CDLL(...).

Anyway, as I suggested above, I think the library handle is created at load/import time, not at new Face.

[rardz]

import freetype as ft to inside the font_process routine give the right running. But, I also found it is somehow fails with import numpy:

import os
import multiprocessing
import numpy

def move_to(a, ctx):
    ctx.append("M {},{}".format(a.x, a.y))

def line_to(a, ctx):
    ctx.append("L {},{}".format(a.x, a.y))

def conic_to(a, b, ctx):
    ctx.append("Q {},{} {},{}".format(a.x, a.y, b.x, b.y))

def cubic_to(a, b, c, ctx):
    ctx.append("C {},{} {},{} {},{}".format(a.x, a.y, b.x, b.y, c.x, c.y))

def font_process(ft_path):

    import freetype as ft
    ft_face = ft.Face(ft_path)
    ft_face.set_char_size( 18*64 )

    ft_face.load_char('B', ft.FT_LOAD_DEFAULT | ft.FT_LOAD_NO_BITMAP)
    ctx = []
    ft_face.glyph.outline.decompose(ctx, move_to=move_to, line_to=line_to, conic_to=conic_to, cubic_to=cubic_to)
    print(f'{os.path.basename(ft_path)} is finished.')

if __name__ == '__main__':
    num_workers = 4
    ft_path_list = ['./a.ttf', './b.ttf', './c.ttf', './d.ttf']

    p = multiprocessing.Pool(num_workers)
    for ft_path in ft_path_list:
        p.apply_async(font_process, args=(ft_path,))
    p.close()
    p.join()

I actually wants to do some matrix calculation with conic_to and cubic_to so come with the import numpy in front of the file. But even only with such a silent line, this example runs blocked.

[HinTak]

Well, that's a numpy problem - they probably have a similar internal design using native libraries (and keeping single handles of those libraries). You'd need to ask them on how to play nice with multiprocessing.

I think this can close as far as freetype-py is concerned; though it might be a good idea to document this limitation / workaround against multiprocessing.

[HinTak]

Have you tried moving import numpy inside each of the routines you want it?

[rardz]

inside routine numpy importing is OK.

[HinTak]

I suspect for numpy, they use multiple native libraries. You 'd be advised to do from numpy import some to only import some you need for each routine.

On second thought, you probably should report the issue to the multiprocessing people and get them to document the issue, since it affects at least two packages which uses native libraries.

[rardz]

OK, thanks a lot.