Threads hanging consuming a full core

[Nemo157]

First noticed 2022-11-14 we had much higher cpu usage than usual

The cratesfyi process was at 300% cpu usage. After restarting the process everything went back to normal.

Again on 2022-11-30 we noticed high cpu usage, this time checking the per-thread stats it was 2 unnamed threads causing it. Again a restart fixed it

I suspected it was related to #1842, so ran a script highlighting every readme from the production database dump, that managed to find one readme that hung at 100% cpu usage. Opened an upstream issue https://github.com/trishume/syntect/issues/460, but this is likely not the only path through which this can be triggered.

[Nemo157]

Looks like there might still be something triggering this, we just got slow requests again and had 4 render threads at high cpu usage

[Nemo157]

Yep, looking through early request failures during this last outage I found one which hangs my local server. Will try and take a look at whether there's something better we can do to avoid this issue.

EDIT: Looking at the specific file a bit more, it's a 42MB single line file, so likely not a complete hang, just taking a really long time to highlight. Will add some limits so we don't try to highlight such large files, and will still consider whether there's something not too complex we could do to have timeouts on highlighting.

[syphar]

Thinking about this one I would close this issue.

There were some safeguards implemented, so the risk of this happening again should be lower.

Of course due to the regex engine, user generated content, and the syntax highlighting engine, we still have a risk of this happening, probably only possible to prevent by using a pool of subprocesses for doing this rendering, where we can kill based on timeouts. Or going back to frontend rendering.