Mock to test mpc when rpc is congested and slow

[ailisp]

Address #642. This PR adds a configurable proxy to the mpc node when talks to lake indexer rpc and when lake indexer talks to s3 (simulate lake being slow)

(Also mentioned in README), After this PR, to run chain-signatures integration test, you will also need toxiproxy. Ensure toxiproxy-server is in PATH

[ailisp]

Great clarification!

Since it is not used in workspaces, interactions from users are not delayed.

Yes interaction from users are not delayed.

Indexers on all nodes are getting new signature requests with a delay

This is not happening, signature requests are from sandbox worker to indexer rpc, which is not delayed. In integration test setup there is only one indexer shared by all nodes.

Sandbox is receiving signature responses with a delay

This happens, but not directly. Mpc is submitting the result to contract on indexer, this is delayed. Then sandbox worker is requesting signature response from indexer, this is not delayed. As a result Sandbox worker is receiving signature response with a delay.

[ailisp]

As discussed with Serhii, I'll also add proxy from lake indexer to s3 to simulate slow down in lake under high load. However proxied s3 (even without any latency rules) seems not working, still trying to figure it out

[ailisp]

@volovyks Now lake indexer -> s3 proxy is added. It is more complicated than expected, as toxiproxy local binary can only proxy local node to container, and toxiproxy container can only proxy container to container, so there are two toxiproxy launched, one local binary running as a subprocess, one docker container. Please take another look

[volovyks]

Looks good! Let's fix the tests and merge it.

Oh, I see, that is the audit.

[github-actions[bot]]

Terraform Feature Environment Destroy (dev-647)

Terraform Initialization ⚙️success

Terraform Destroy success

Show Destroy Plan

``` No changes. No objects need to be destroyed. Either you have not created any objects yet or the existing objects were already deleted outside of Terraform. Destroy complete! Resources: 0 destroyed. ```

Pusher: @ailisp, Action: pull_request, Working Directory: `, Workflow:Terraform Feature Env (Destroy)`