tswaters
commented
7 years ago Bit of a follow up here.... I found the two things that were causing our immediate problem of being unable to boot all the services up. First, in the front-end service. We had something along the lines of the following that appeared in the startup of the application:
module.exports = app => new Promise((resolve, reject) => {
const router = new Router()
glob('some-files', (err, files) => {
if (err) { return reject(err) }
files.forEach(file => router.use(require(file)(app)))
resolve(router)
})
}Changing it like so:
const Route1 = require('./some-path')
const Route2 = require('./some-path')
const Route3 = require('./some-path')
module.exports = app => new Promise((resolve, reject) => {
const router = new Router()
router.use(Route1(app))
router.use(Route2(app))
router.use(Route3(app))
resolve(router)
})made it function properly again. My guess is that the require call here is blocking, and while the event loop is blocked, it can't respond to swim in a timely fashion. As more time progresses in a blocked state, the more likely pins under this service are pinged by swim and marked as faulty.
I still not entirely sure why it goes into a loop once a certain number of pins are reached in a service though. Perhaps the added traffic / processing around marking nodes as failed and re-added is itself blocking response to pings, so things are marked as faulty, further propagating the problem as each service needs to receive the message that it needs to remove the node.
The second was with how the pins were registered, before we might have had something like this:
const pins = [
{pin: 'domain:someDomain,cmd:someCmd1'},
{pin: 'domain:someDomain,cmd:someCmd2'},
{pin: 'domain:someDomain,cmd:someCmd3'},
{pin: 'domain:someDomain,cmd:someCmd4'},
]Making it like this:
const pins = [
{pin: 'domain:someDomain,cmd:*'}
]Makes it considerably faster to not only spin up, but and I figure helps towards limiting the number of things that swim needs to manage and the resulting traffic storm when a new service is either added or removed removed from the mesh.
ghost
otaviosoares
MikeLindenau
rjrodger
vforv
danielo515
Apologies, this a bit long but is the culmination of quite a lot of pain & struggling and debugging with a pretty fatal problem.
--
We've got a bit of a problem, I've been trying myself to debug it but I'm basically at wit's end trying to figure out what the problem is. I have a suspicion it relates to too many pins on mesh and timing issues when a service is actually ready to receive ping requests from swim and those pings failing because it's not ready.
When we started seeing this, the symptoms were that services in the mesh would lose connections. Specifically, we have an
authservice that wrapsseneca-userand on the front end, an auth pattern is hit on almost every page visit. We would see the front-end service unable to find the auth pattern and a failure withact_not_found. We're still under development so the resolution is to kill everything & start it up again.The hope was that this was an isolated thing that we would only see during development - but now we're looking to get this running outside of local developers machines, using docker and rancher to spin up all the microservices and we're seeing it there now, when upgrading nodes to the latest version (basically stop, pull latest image, start) and when scaling to multiple instances.
I discovered I could clearly see the problem if I pass the following option on the base service,
With this present, we can see when nodes are added and when they are removed. When the symptoms above show things go, for lack of a better word - haywire. This log goes insane with a few messages per second, both add and remove showing pins from all microservices in the mesh.
It spreads like a virus it seems - starting with one node and spreading to others. When it gets in this state, the only way I've seen to resolve it is to kill everything and start it up. When this happens locally running everything (10 microservices), the system slows to a crawl... in rancher, nodes go unhealthy, are removed & re-added and it's basically a perpetual reboot loop.
Our sample configuration for a service looks something like the following:
The liberal use of IP_ADDRESS is to get it working with docker/rancher. The base node is the only one to use a host name because (for now) there is only one of them. The networking is handled by rancher here and we get access to related services by host name. In practice, this falls down when there are multiple services all using the same name, it appends numbers... so we just use IP address in the config
I've omitted pins, but in practice on each service there are 4
model:observepins for cache clears and service startup notifications, and each service exposes ~7-9 pins in addition to this. The front-end one has an addedmodel:observeforroute:setfor seneca-web. Each service has a health check setup with a pre-defined port - rancher uses this to perform health checks.Other notes and things I've noticed in debugging this (fruitlessly):
if a service calls into another while it is booting up, things are far more likely to break. We got around this locally by a liberal usage of
timeoutin the fuge config - and in rancher, by a clever startup script that will wait for the healthcheck of dependant services to be available before attempting to start.the
model:observestrangely seems to cause the problem more often. After we added service number 10, we weren't able to start up the project at all without hitting this. On the front-end service after receiving the route payload from seneca-web (3 services emit their own routes, including number 10) we'd see a remove/add for pins on the front-end. It seems the cache model:observe would cause it to spin out of control here... commenting those out, we can at least get it started... it still removes/adds route:set and the healthcheck, but it doesn't spin out of control.looking through the callstack on break points, it seems that nodes are marked as faulty by swim, make their way to sneeze and finally to mesh. Why this is happening is still an unknown to me and may be the underlying cause of the problem - the services should be alive and kicking by the time swim starts polling them, right?
we are running with
seneca.fixedargs.fatal$ = falseon all services. This is so we can pass back errors in seneca actions without killing the services. I tried removing this to see if things would die properly but this was not the case. There are no errors raised throughout this, just a lot of network traffic and degraded nodes.--
I have a feeling this can be worked around by providing options to swim by way of sneeze options, but for the life of me, I don't know what the best options are, nor do I know if this is just a stopgap until we add more pins/services and need to increase timeouts again.