Fauntleroy
commented
10 years ago Workers are instantiated by this method: https://github.com/SparkartGroupInc/solidus/blob/master/lib/preprocessor.js#L52-L59
Preprocessor workers are the only child processes that exist in Solidus. These processes are also automatically reaped and restarted when they encounter an error or hang. It's possible there may be some problem with https://github.com/rvagg/node-worker-farm that causes processes not to be properly reaped. I'm also not completely sure if the graphs Modulus is giving us keep track of resource usage in child processes, so there's a possibility they might be locking the instance up. Presently workers can only take 100 calls before they're automatically reaped and restarted.
I'm still convinced that the problem is not within Solidus itself, due to the fact that this happens across all of our Solidus sites at once. All of our sites share other common threads; using the Storyteller.io proxy, the same DNS service, running on Modulus, fronted by Edgecast, etc etc. Isn't it more likely the problem lies in one of those places?
joanniclaborde
pushred
There’s been a recurring issue where resource requests suddenly start to fail across all of our sites hosted with Modulus. This usually triggers a period of downtime due to very high responses times. The most recent incident occurred on March 26, lasting for an hour around 9:30-10:30. Attempts to restart the Modulus servers, re-deploy, etc. didn’t have an effect. Recovery usually happens on it’s own, quickly and suddenly. Possibly when Modulus kills processes that may be at fault. In this incident they were not seeing anything unusual on their end.
Pingdom looked like:
Modulus looked like:
I can provide the full logs on request. I believe the
ETIMEDOUTerror was the core issue, with the high page response times and preprocessor errors being symptomatic.Storyteller.io
Resources on sites are generally proxied via Storyteller.io. Modulus suspected issues on our end, but checking our own systems nothing seemed to be wrong. Heroku reported no issues for the day or the ~2 days around it. Pingdom records 100% uptime for the period. Logs from the period showed no requests for site-related resources, starting around 9:36pm which correlates to the longest period of downtime recorded by Pingdom (36m):
The last request from a site was here:
Requests from the
hipster-toolsapp for resources used there continued however:Random requests from Pingdom were also successful:
The next successful request from a site comes here:
Pingdom was making successful requests around this time before another downtime event of 18 minutes starting at 05:13 and lasting until 05:31. The site remained operational for another 15 hours from there, with a 1 minute flicker, and then 11+ days of uptime since.
If this was an intermittent network issue that could explain why our app’s requests continued to work while outside requests did not. The Pingdom checks of our status endpoint aren’t quite the same, but they definitely don’t support faulting Storyteller.io in these issues.
Child Processes
There’s something of a pattern to this happening in the late night/early morning hours. The incident prior occurred March 12 from 12:19am with non-stop flapping all the way overnight until 8am, totaling 5 hours 43 minutes. Before this February 28 had 43 minutes of downtime around 11:40-12:26am. Another pattern are long periods of 1-5 minute flapping that culminate in longer periods of downtime that then suddenly recover. Timestamps can be seen from Pingdom.
In the case of March 12 Modulus was experiencing an issue where a percentage of servos failed to start due to EADDRINUSE error, so re-deploying wasn’t even possible without their intervention. They had this to say.
It’s possible this incident was unrelated, but it does make me wonder about the child processes we spawn. Charlie previously showed us some examples of zombie processes they saw spawned from Solidus servers. Haven’t heard of further incidents though. This may have predated the switch to node-worker-farm.
Are any child processes spawned for resource requests?
Resiliency
We may not be able to identify the exact cause of this issue if it is truly intermittent network latency. But if this is the case I think we need more resilient resource fetching that doesn’t bring an entire site down. In Orator a site continued to load if resource requests failed, leading to issues of disappearing content and such but otherwise maintained some level of uptime.
There is a 3.5 second timeout for resources, but is the fetching synchronous? I’m wondering why the rest of the site doesn’t respond if all the resources timeout around the same time.