Investigate occasional high CPU usage of vault clients

[ebma]

It seems like sometimes, the vault clients' CPU usage increases up to 100% without any apparent reason. This leads to the vault client becoming unresponsive and not handling bridge requests anymore.

What we know:

• This issue happens across all Amplitude and Pendulum vaults at around the same time.
• In the monitoring, we can see that the CPU usage increases from 50% to 100% within 45 minutes.
• Memory and disk usage are not increasing during this time window.
• After the periodic restart, the CPU usage goes back to ~50%
• There are no messages appearing in the logs (at least on 'INFO' log level).
• The Spacewalk-related metrics are unavailable for this timeframe which indicates that the whole client becomes unresponsive

[ebma]

@zoveress to facilitate the investigation of this ticket, can we change the log level of one of the Spacewalk clients to 'DEBUG'? Doesn't matter which one on Amplitude or Pendulum as they all seem to have the same issue at the same time (at least when it occurs for which we don't know the reason yet).

[ebma]

@pendulum-chain/product this is a bug that can lead to the vault client being unresponsive and only able to bridge after the next periodic restart. It does not occur frequently and we first have to monitor the instances when this is happening.

[b-yap]

From the #DevOps Incidents, it started as far back as Sept 17, 2023.

extra: Memory Usage High alerts happened on Sept 21, 2023

[vadaynujra]

@annatekl anything blocking this from being moved to Ready?

[annatekl]

moved @vadaynujra

[ebma]

@b-yap please don't forget to assign yourself to tickets once you start working on them so that others don't. 😬

[b-yap]

@ebma noted.

Question; as this has something to do with the vaults, should we move this ticket over there?

[ebma]

Over where? It's already in the spacewalk repository or would you prefer to have it elsewhere? 😅

[b-yap]

ah nevermind, I got confused with the zenhub "Pendulum" icon :weary: