awolden / brakes

Hystrix compliant Node.js Circuit Breaker Library
MIT License
300 stars 35 forks source link

CPU growth #88

Open bdfinst opened 7 years ago

bdfinst commented 7 years ago

Using the default setting, we experienced growth in CPU usage and increasing performance degradation throughout the day. We were forced to add a cron job to bounce the service every 6 hours. When I removed brakes, CPU usage returned to normal.

awolden commented 7 years ago

Hi @yamatoman,

Sorry to hear you are having troubles with Brakes. I haven't seen any rise in CPU usage using brakes in my services, so I would be interested to know more about how it is being implemented in your application. Can you provide some more details on how you are implementing it?

Thanks,

-Alex

awolden commented 6 years ago

@yamatoman Hopefully this message finds you well. I was able to reproduce a very similar issue by instantiating new brakes instances in the request handler for a service. This was causing run away CPU growth as a new circuit breaker instance was being created for every request. I'm not sure if this was causing your issue, but I hope the information might be helpful.

bdfinst commented 6 years ago

Sorry I've not gotten back to you. Buried in deliverables. Let me check how we have it implemented and see if it's similar. Thanks so much for the follow up!

-- Bryan


From: Alexander Wolden notifications@github.com Sent: Thursday, July 27, 2017 3:52:32 PM To: awolden/brakes Cc: Bryan Finster; Mention Subject: Re: [awolden/brakes] CPU growth (#88)

@yamatomanhttps://github.com/yamatoman Hopefully this message finds you well. I was able to reproduce a very similar issue by instantiating new brakes instances in the request handler for a service. This was causing run away CPU growth as a new circuit breaker instance was being created for every request. I'm not sure if this was causing your issue, but I hope the information might be helpful.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/awolden/brakes/issues/88#issuecomment-318483087, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH0MN_ah1mVjWaZ_qPFvUWVqP40XvCAIks5sSPiQgaJpZM4OWNnA.

bdfinst commented 6 years ago

Here's how we're implementing

function circuitBreakify(fnToWrap) {
    const brake = new brakes(fnToWrap);
    return brake;
}
.
.
.
function getInfoForThing(req, thing) {
    const request = httpHelper.circuitBreakify(getInfoForThingRequest);
    return request.exec(req, thing);
    return getInfoForThingRequest(req, thing);
}

So, we're doing exactly what you said. How would you recommend we implement instead?

Thanks!

awolden commented 6 years ago

You will want to create 1 brakes instance for every function you want to circuit break, and stick it in a reference-able spot.

Something like this:

// create each circuit exactly once some where else in your app or module
const circuits = {
  getInfoForThingRequest: new Brakes(fnToWrap);
}

...
...

function getInfoForThing(req, thing) {
    // do not create a circuit per request, instead exec a previously instantiated circuit.
    return circuits.getInfoForThingRequest.exec(req, thing);
}
bdfinst commented 6 years ago

Thanks, let me give that a try.

-- Bryan


From: Alexander Wolden notifications@github.com Sent: Friday, July 28, 2017 11:33:00 AM To: awolden/brakes Cc: Bryan Finster; Mention Subject: Re: [awolden/brakes] CPU growth (#88)

You will want to create 1 brakes instance for every function you wan't to circuit break, and stick it a reference-able spot.

Something like this:

// create each circuit exactly once some where else in your app or module const circuits = { getInfoForThingRequest: new Brakes(fnToWrap); }

... ...

function getInfoForThing(req, thing) { // do not create a circuit per request, instead exec a previously instantiated circuit. return circuits.getInfoForThingRequest.exec(req, thing); }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/awolden/brakes/issues/88#issuecomment-318701267, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH0MN4PQ_NFvpAQndUHEGUuMRgFvt1KLks5sSg08gaJpZM4OWNnA.