alakajam-team / alakajam

Website powering the Alakajam! game making community
https://alakajam.com/
Other
28 stars 13 forks source link

[⌛ Resolved?] Unknown infinite loop #586

Closed mkalam-alami closed 3 years ago

mkalam-alami commented 3 years ago

A 100% CPU issue making the server inoperable has been encountered multiple times before and during the 10th Alakajam. I suspect this is related to templating, as the biggest update introduced since the previous jam is the JSX templating system.

mkalam-alami commented 3 years ago

Happened again on Oct 5th 22:35 UTC, it locked the full server despite the lower CPU priority given to the Node process

mkalam-alami commented 3 years ago

Apache access logs haven't yielded much, here's the last few requested URLs before the 22:35 crash :

[05/Oct/2020:20:43:46 +0000] "GET /post/1380/i-get-the-feeling-well-all-be-seasick-by-the-end-of-the-rating-phase/ HTTP/1.1"
[05/Oct/2020:20:43:47 +0000] "GET /scorespace-alakajam HTTP/1.1"
[05/Oct/2020:20:43:47 +0000] "GET /post/1380/i-get-the-feeling-well-all-be-seasick-by-the-end-of-the-rating-phase HTTP/1.1" 
[05/Oct/2020:20:43:47 +0000] "GET /posts?p=10 HTTP/1.1" 
[05/Oct/2020:20:43:48 +0000] "GET /robots.txt HTTP/1.1"
[05/Oct/2020:21:13:36 +0000] "GET /robots.txt HTTP/1.1"
[05/Oct/2020:21:13:36 +0000] "GET /robots.txt HTTP/1.1"
[05/Oct/2020:22:19:28 +0000] "GET /robots.txt HTTP/1.1" 
[05/Oct/2020:22:19:28 +0000] "GET /robots.txt HTTP/1.1" 

The few Node server logs in that timeframe show nothing special, except a 500 error 40mn before the freeze (now fixed with 380563b).

I have set up per-request server logging just in case it could help, but the crash seems to be more subtle than just something route-specific.

mkalam-alami commented 3 years ago

With my jsx-pistols lib remaining the main suspect, I have optimized its behavior in production mode (instead of a transpiler, it basically becomes a pass-through that only loads the compiled templates using NodeJS' require()). It should reduce its use of resources a lot. I have good hopes it will help.

mkalam-alami commented 3 years ago

Assuming fixed 🤞