When one single request failed during the parallelization, it left the memoized hydra in it's current state. Because every single request in the parallelization calls back, and when one of them raises an error through LHC the others will not continue, and worse, will leave the memoized hydra in it's current state.
https://github.com/local-ch/lhc/blob/master/lib/lhc/request.rb#L154
The next request parallelisation picked up the memoized hydra in it's broken state, and when it made requests to a url that was already part of the broken memoized hydra, somehow ethon due to the way they use memory pointers failed fast with the execution of those requests entirely.
https://github.com/typhoeus/ethon/blob/master/lib/ethon/multi/operations.rb#L26
It's an endless circle of Status 0 (timeouts) being served from memory (without even making the requests). That's also why neither Backend nor Devops ever seen the traffic going out of that pod, nor ending up on the BE systems.
Because hydras are memoized by thread, this explains also why applications after a recycle work and then after time start to fall into this pattern again, because each thread's memoized hydra needs to built up those broken queues first. Then when all threads are suffering from this issue, the application because unresponsive.
dimameshcharakou
commented
4 years ago
This is a big one.
For those of you that know, but some of our applications are constantly suffering from
Status 0 (timeout)responses.Especially in AMA where there was nothing else left to do, but recycling the pods, in order to get the application stable again.
What I think was happening
LHC uses
Typhoeus::Hydrato parallelize requests. https://github.com/typhoeus/typhoeus/blob/master/lib/typhoeus/hydra.rb#L67LHC used the memorized instance of Hydra.
When one single request failed during the parallelization, it left the memoized hydra in it's current state. Because every single request in the parallelization calls back, and when one of them raises an error through LHC the others will not continue, and worse, will leave the memoized hydra in it's current state. https://github.com/local-ch/lhc/blob/master/lib/lhc/request.rb#L154
The next request parallelisation picked up the memoized hydra in it's broken state, and when it made requests to a url that was already part of the broken memoized hydra, somehow ethon due to the way they use memory pointers failed fast with the execution of those requests entirely. https://github.com/typhoeus/ethon/blob/master/lib/ethon/multi/operations.rb#L26
It's an endless circle of Status 0 (timeouts) being served from memory (without even making the requests). That's also why neither Backend nor Devops ever seen the traffic going out of that pod, nor ending up on the BE systems.
Because hydras are memoized by thread, this explains also why applications after a recycle work and then after time start to fall into this pattern again, because each thread's memoized hydra needs to built up those broken queues first. Then when all threads are suffering from this issue, the application because unresponsive.