bobek commented 1 year ago

Context

No response

Bug description

We have been facing random SIGBUS crashes after update to 9.1.12-insiders-4.34.0.

INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /tmp/pom
[1]    28073 bus error (core dumped)  mkdocs build -d /tmp/pom --strict --config-file mkdocs_insiders.yml

Way to replicate is to delete .cache directory to force recreation of all social cards. It will then crash with about 75% chance.

5515 inspired me to try setting `concurrency: 2`, I cannot say it fixed the issues, but I have not seen it crash so far. I have not studied the code yet, but maybe the `cpus-1` doesn't work (yet) and your fix from #5515 will basically fix this one as well?

I have Python 3.11.2, but also tested on 3.10 with the same behavior.

gdb backtrace from core file:

#0  0x00007fa828822d4e in ?? ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#1  0x00007fa82881ca6c in ?? ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#2  0x00007fa828866de5 in ?? ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#3  0x00007fa82886c083 in ?? ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#4  0x00007fa828864608 in hb_face_create ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#5  0x00007fa8289ab1ec in hb_ft_face_create ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#6  0x00007fa8289ab355 in hb_ft_font_create ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#7  0x00007fa8289ab684 in hb_ft_font_create_referenced ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/../Pillow.libs/libharfbuzz-3543f599.so.0.60710.0
#8  0x00007fa829812344 in raqm_layout ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/_imagingft.cpython-311-x86_64-linux-gnu.so
#9  0x00007fa82980dafb in ?? ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/_imagingft.cpython-311-x86_64-linux-gnu.so
#10 0x00007fa82980dead in ?? ()
   from /home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/_imagingft.cpython-311-x86_64-linux-gnu.so
#11 0x00000000005230d0 in ?? ()
#12 0x000000000053ac2c in PyObject_Vectorcall ()
#13 0x000000000052b940 in _PyEval_EvalFrameDefault ()
#14 0x00000000005855a4 in ?? ()
#15 0x0000000000585174 in ?? ()
#16 0x000000000056a381 in PyObject_Call ()
#17 0x000000000052f8a2 in _PyEval_EvalFrameDefault ()
#18 0x00000000005855a4 in ?? ()
#19 0x000000000058510e in ?? ()
#20 0x000000000052f8a2 in _PyEval_EvalFrameDefault ()
#21 0x000000000055c931 in _PyFunction_Vectorcall ()
#22 0x000000000052f8a2 in _PyEval_EvalFrameDefault ()
#23 0x00000000005855a4 in ?? ()
#24 0x0000000000585148 in ?? ()
#25 0x000000000067917c in ?? ()
#26 0x0000000000653cb4 in ?? ()
#27 0x00007fa82bafefd4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#28 0x00007fa82bb7f5bc in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Reproduction

I don't have reproduction archive at the moment. Our code-base is fortunately open ( https://github.com/ipfabric/docs )

Steps to reproduce

remove .cache
run mkdocs build to get all social cards generated

  - social:
      cards_layout: default/variant
      concurrency: 2
      cards_layout_options:
        background_color: "#264183"
        color: "#FFFFFF"

Browser

No response

Before submitting

[X] I have read and followed the bug reporting guidelines.
[X] I have attached links to the documentation, and possibly related issues and discussions.
[X] I assure that I have removed all customizations before submitting this bug report.
[X] I have attached a .zip file with a minimal reproduction.

squidfunk commented 1 year ago

Thanks for reporting. Could you please try the tip of the master branch of Insiders? If setting concurrency: 2 on the latest release, it's very likely related to the problem reported in #5515 and should be fixed subsequently.

squidfunk commented 1 year ago

... also, if 4.33.2 works, it's very likely related to the problem reported in #5515.

bobek commented 1 year ago

I have just tested the tip of the master with 1f0db88ae applied. Also verified, that it really passes expected value to the ThreadPoolExecutor. Everything seems fine.

Interestingly at about 10% of executions, I would get the following crash:

ERROR    -  Error building page 'Getting_Started/Overview/faq.md': unknown file format
Traceback (most recent call last):
  File "/home/bobek/Sources/ipf/docs/docs/venv/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/mkdocs/__main__.py", line 250, in build_command
    build.build(cfg, dirty=not clean)
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/mkdocs/commands/build.py", line 329, in build
    _build_page(file.page, config, doc_files, nav, env, dirty)
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/mkdocs/commands/build.py", line 234, in _build_page
    output = config.plugins.run_event('post_page', output, page=page, config=config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/mkdocs/plugins.py", line 520, in run_event
    result = method(item, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/material/plugins/social/plugin.py", line 162, in on_post_page
    raise future.exception()
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/material/plugins/social/plugin.py", line 297, in _generate
    self.card_layer_jobs[h].result(),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/material/plugins/social/plugin.py", line 331, in _render
    image = self._render_typography(layer, image)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/material/plugins/social/plugin.py", line 416, in _render_typography
    current, spacing = _metrics(path, line, input)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/material/plugins/social/plugin.py", line 882, in _metrics
    typeface = ImageFont.truetype(path, 1000)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/ImageFont.py", line 996, in truetype
    return freetype(font)
           ^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/ImageFont.py", line 993, in freetype
    return FreeTypeFont(font, size, index, encoding, layout_engine)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bobek/Sources/ipf/docs/docs/venv/lib/python3.11/site-packages/PIL/ImageFont.py", line 248, in __init__
    self.font = core.getfont(
                ^^^^^^^^^^^^^
OSError: unknown file format

squidfunk commented 1 year ago

Hmm, interesting. Are there actually font files in .cache/plugins/social/fonts? Fonts are downloaded from Google Fonts, maybe that failed for some reasons and it's masquerades by this error. Could you try and create a reproduction?

Additionally, does the same happen when you run with concurrency: 1? Because before you said that setting it to concurrency: 2 fixed the problem entirely, which is what was fixed in the last commit mentioned.

squidfunk commented 1 year ago

Okay, I think I know where this might be coming from – currently, if the font that is used for the social card hasn't been downloaded before, a thread will download it from Google. However, all threads might be trying to download it from Google and will overwrite the font files multiple times. This might lead to a half-overwritten font file that is then used by a thread that downloaded it earlier, resulting in a broken font file.

This is now fixed in 99fb7c16c. Downloading of fonts is now synchronized with a lock, so the font is only downloaded once. @bobek could you check the tip of master again, and check whether this fixes OSError: unknown file format?

bobek commented 1 year ago

Sorry, @squidfunk , was snowed under daily stuff.

This might lead to a half-overwritten font file that is then used by a thread that downloaded it earlier, resulting in a broken font file.

Sounds plausible, those pesky race conditions ;)

@bobek could you check the tip of master again, and check whether this fixes OSError: unknown file format?

Confirmed, run through the sequence without hitting any issues.

Thank you!

squidfunk commented 1 year ago

Perfect! Glad we could sort it out that quickly ☺️ I'll issue a new Insiders release today.

squidfunk commented 1 year ago

Released as part of 9.1.13+insiders-4.34.1.

squidfunk / mkdocs-material

SIGBUS while rendering social cards #5521

Context

Bug description

5515 inspired me to try setting `concurrency: 2`, I cannot say it fixed the issues, but I have not seen it crash so far. I have not studied the code yet, but maybe the `cpus-1` doesn't work (yet) and your fix from #5515 will basically fix this one as well?

Related links

5515

Reproduction

Steps to reproduce

Browser

Before submitting

squidfunk / mkdocs-material

SIGBUS while rendering social cards #5521

Context

Bug description

5515 inspired me to try setting concurrency: 2, I cannot say it fixed the issues, but I have not seen it crash so far. I have not studied the code yet, but maybe the cpus-1 doesn't work (yet) and your fix from #5515 will basically fix this one as well?

Related links

5515

Reproduction

Steps to reproduce

Browser

Before submitting

5515 inspired me to try setting `concurrency: 2`, I cannot say it fixed the issues, but I have not seen it crash so far. I have not studied the code yet, but maybe the `cpus-1` doesn't work (yet) and your fix from #5515 will basically fix this one as well?