Create service for periodic Spacewalk vault client integration tests

[ebma]

In order to make sure that our vault client infrastructure and bridging work as expected, we should create a service that periodically bridges assets from and to Stellar. If these processes don't finish in a given period of time, e.g. 1 hour, the service sends alert messages to the #devops-incidents slack channel.

How it works

The logic is the following. For every network in NETWORKS: for every vault in TESTED_VAULT_IDS:

• Issuing assets
  • The service creates a new issue request for BRIDGED_AMOUNT amount of the vaults-wrapped assets
  • It executes the expected payment transaction on Stellar (with the matching amount, asset, memo, and destination) using the account for either STELLAR_ACCOUNT_SECRET_MAINNET or STELLAR_ACCOUNT_SECRET_TESTNET depending on the stellar_mainnet parameter of the configuration
  • It monitors the on-chain events and waits for up to COMPLETION_WINDOW_MINUTES minutes for the vault to fulfill the issue request (ie. a matching Issue.Execute event is emitted)
  • If this event is not emitted in the given number of minutes, the service considers the bridging to be failed and sends a message to a channel on slack, describing the problem and mentioning the ID of the vault that failed to execute the issue request.
• Redeeming assets
  • To be 100% sure that the service can redeem assets on a vault client, we should only test redeeming assets after a successful test for issuing assets. Otherwise, we might not be able to redeem the desired amount of assets.
  • The service creates a new redeem request for BRIDGED_AMOUNT amount of the vaults-wrapped assets, specifying the (raw 32-bytes) Stellar public key of STELLAR_ACCOUNT_SECRET as the destination
  • It monitors the on-chain events and waits for up to COMPLETION_WINDOW_MINUTES minutes for the vault to fulfill the redeem request (ie. a matching Redeem.Execute event is emitted)
  • If this event is not emitted in the given number of minutes, the service considers the bridging to be failed and sends a message to a channel on slack, describing the problem and mentioning the ID of the vault that failed to execute the redeem request.
• Wait for TEST_DELAY_INTERVAL_MINUTES and repeat

Sending messages to the 'devops-incidents' channel on slack

We will use an incoming webhook to post messages to this channel on Slack. The URL is https://hooks.slack.com/services/T02G2ERGQ/B04BPP5J6RH/dQbJ1L3W4zLWtOLpFCeC14q. However, we need to be careful about sharing this URL and thus should feed the ID with an environment variable (with the ID being T02G2ERGQ/B04BPP5J6RH/dQbJ1L3W4zLWtOLpFCeC14q in this case).

Parameters

We have the following set of configurable parameters:

• COMPLETION_WINDOW_MINUTES - the number of minutes a vault client has the chance to fulfill issue/redeem requests before the service considers them as failed. Default is 60 minutes
• BRIDGED_AMOUNT - the amount of wrapped assets that is to be used for created issue/redeem requests. Default is 100000000000 (ie. 0.1 'units').
• STELLAR_ACCOUNT_SECRET_MAINNET and STELLAR_ACCOUNT_SECRET_TESTNET - the secret key of the Stellar account that will be used by the service. It needs to hold at least BRIDGED_AMOUNT of tokens of each Stellar asset that is supposed to be tested. Mandatory parameter without default value
• TEST_DELAY_INTERVAL_MINUTES - this defines the number of minutes that the service will wait before starting another round of testing. Default is 720 minutes.
• NETWORKS - an array of settings for each Spacewalk-supported network that we want to test
  • WSS - the URI of the Spacewalk-chain we want to connect to
  • STELLAR_MAINNET - boolean parameter whether to use Stellar mainnet or testnet.
  • TESTED_VAULT_IDS - an array of the IDs of the vaults that are supposed to be tested

To make it easier to set a configuration per network, we want to have a configuration file that should look roughly as follows:

{
  "completion_window_minutes": 60,
  "bridged_amount": 1000000000000,
  "test_delay_interval_minutes": 720,
  "networks": [
    {
      "name": "foucoco",
      "wss": "wss://rpc-foucoco.pendulumchain.tech",
      "stellar_mainnet": false,
      "tested_vault_ids": [
        "6asdf",
        "6asdf"
      ]
    },
    {
      "name": "amplitude",
      "wss": "wss://rpc-amplitude.pendulumchain.tech",
      "stellar_mainnet": true,
      "tested_vault_ids": [
        "6asdf",
        "6asdf"
      ]
    }
  ]
}

the STELLAR_ACCOUNT_SECRET_MAINNET and STELLAR_ACCOUNT_SECRET_TESTNET parameters should be passed as an environment variable.

Deployment

I would suggest we deploy this in a way that is similar to our token-stats API service i.e. on an extra server that runs 24/7.

[ebma]

@TorstenStueber could you please add the webhook URL for the #devops-incidents channel to the description?

[annatekl]

Hey team! Please add your planning poker estimate with Zenhub @adelarja @b-yap @ebma @gianfra-t @TorstenStueber

[TorstenStueber]

Ignore this comments, see below

@ebma an example call to the webhook is (just try it)

curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/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

This is of course sensitive information and must not end up in a public repository – best is probably to manage this as a deployment secret.

[TorstenStueber]

Great initiative @ebma . Is your intention that this be written in Rust or TypeScript?

[ebma]

I'd prefer if we implement this service in TypeScript because I assume the implementation would be faster and easier to deploy.

[TorstenStueber]

The correct webhook for "Devops Incidents" is https://hooks.slack.com/services/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

[annatekl]

@ebma does this require runtime upgrade?

[ebma]

@annatekl no it does not. Generally, you can assume that only PRs created in the pendulum repository require a runtime upgrade to take effect.