Closed b-yap closed 3 months ago
@ebma Would be great if we have the vaults running with this feature. I bet it'll look very different (and interesting).
True, but I'm not sure if we can easily connect to the clients with tokio-console
if they are running in a docker container.
@zoveress in this PR we added support for a profiling tool that lets us investigate resource consumption of tasks within our process. As you can see here, it's fairly simple to run. As far as I understand, the changes we made to the client will make it expose a new endpoint (by default) on port 6669 that is used by the tokio-console
tool for accessing the debug information. Using this setup would be very easy to do in the EC2 instances but maybe not in our Kubernetes cluster. @zoveress do you have an idea how we could be able to access it also on Kubernetes?
This is not super important but would be nice to have.
Is the endpoint password secured?
No it's not and I think it's only supposed to be used locally. Can we expose it to the GitLab runner only and access it from there somehow?
This does not completely close #512, but it helps with tracking.
But if we can see which threads have been running the longest; which ones are idle for a long time, it could help determine where/when a vault gets stuck.
The
tokio-console
is described as:^ From using tokio-console, a zombie task was found. When the vault restarts, it shuts down all running tasks except for 1: https://github.com/pendulum-chain/spacewalk/blob/e7a672ba1e21c98a70df30a6ee458317951dd597/clients/wallet/src/resubmissions.rs#L40-L47
Aside from this, I found out that the stream also gets stuck: https://github.com/pendulum-chain/spacewalk/blob/e7a672ba1e21c98a70df30a6ee458317951dd597/clients/stellar-relay-lib/src/connection/connector/message_reader.rs#L82 Requires a timeout to trigger reconnection.
How to begin the review:
I have added comments to help with the review.
tracing
feature oftokio
is necessary fortokio-console
console-subscriber
dependency is necessary fortokio-console
StellarWallet
:fn stop_periodic_resubmission_of_transactions(&mut self)
fn start_periodic_resubmission_of_transactions_from_cache(...)
:StellarWallet
's _resubmission_end_signal
_ field.clients/vault/src/system.rs:
fn stop_periodic_resubmission_of_transactions(...)
from thewallet
field ofVaultService
:calls the
fn shutdown_wallet(...)
when the service stops.READ_TIMEOUT_IN_SECS
; how long to wait on reading from the stream.tokio-console