Open mikael-carlstedt opened 7 months ago
I can add that the toxicity that we are trying to remove is a 6 s downstream delay on all requests, if that could be a clue to the root cause.
An observation of interest is that the message "Interrupting the previous toxic to update its output" is missing for the nine first links listed with the message "Waiting to update links", which would seem to point at the tenth link never gets completed (it appears to start with the last link and iterate backwards in the list).
Could this be the same issue as described in #427 ?
We sometimes observe that Toxiproxy becomes unresponsive when we try to delete a toxic and a server restart is required to recover. This is what it looks like in the server side logs:
Compared with a previous successful delete toxic operation, this one is missing the terminating log line:
Assuming that is when it releases the mentioned lock, and the reason why the subsequent retries from the client are stuck on acquiring the lock.
The log excerpt has been cleaned from proxy traffic logging.