Feature request: chaos-anvil

prestwich commented 1 year ago

Component

Anvil

Describe the feature you would like

Description

Add chaos modes to anvil to better simulate real network conditions.

Motivation

RPC issues are common causes of bugs in foundry, script execution, and off-chain actors. Because mainnet nodes are consistently less reliable than testnet nodes and simulation nodes, it is difficult to test resilience to RPC issues in an application before production deployment

To support stress testing libs, applications, etc, we should add chaos features to anvil, to simulate faulty or weird behavior of mainnet RPC nodes

Details

A --chaos flag for anvil that takes the following instructions each of which enables a specific feature:

tx - getTransaction may not fail return a known transaction
send - txns broadcast do not always confirm immediately. They may confirm after a delay (which should not block RPC), or not at all.
getLog - events are sometimes randomly deleted from eth_getLogs request responses
streamLog - events are sometimes randomly dropped from log streams

Example invocation anvil --chaos tx,send,getLog

Bonus points if we can configure frequency at which each of these events occur :)

Additional context

No response

gakonst commented 1 year ago

@mattsse this would be really sick. fault injection effectively. it's possible we could add this feature cleanly by implementing it as a wrapper around our current anvil axum api? that would be minimally invasive while letting us Config everything that the user sees.

I wonder if a generalized version of this is a proxy queueimplementation that sits in front of any node?

mattsse commented 1 year ago

ye, we could easily intercept this here:

https://github.com/foundry-rs/foundry/blob/07dccfd6ebac583ccd63bb818bf27e29b2a4005f/anvil/src/eth/api.rs#L138

with something like:

fn intercept(&self, req: EthRequest) -> Result<EthRequest, ResponseResult>;

0xcacti commented 1 year ago

If there is still interest in this feature, I stubbed out a super basic outline for how this might work. You can see it at my fork here - https://github.com/0xcacti/foundry/tree/0xcacti/chaos/crates/anvil

Basically, you add a config flag --chaos to the cli. You can pass in --chaos tx,send for some default rate of fault injection. Or you can configure probabilities with --chaos tx:0.005, ... and tx will inject faults 0.5% of the time.

If I am not totally off base here, I would need guidance on determining exactly which api methods we want to intercept and exactly how we want the failures to occur. Of course, no concerns if I am not going down the right path and we want to scrap this.

I also wrote out an example for how we might intercept logs It's pretty verbose and I am sure there is a better way to do it, but it's a start. (Again, this might not even be the right behavior we want)

    /// Returns logs matching given filter object.
    ///
    /// Handler for ETH RPC call: `eth_getLogs`
    pub async fn logs(&self, filter: Filter) -> Result<Vec<Log>> {
        node_info!("eth_getLogs");
        let logs = self.backend.logs(filter).await;
        if let Some(chaos_config) = &self.chaos_config {
            if let Some(failure_rate) = chaos_config.get_log {
                if rand::random::<f64>() < failure_rate {
                    if let Ok(logs) = logs {
                        return Ok(self.intercept_logs(logs, failure_rate))
                    }
                }
            }
        }
        logs
    }

    fn intercept_logs(&self, logs: Vec<Log>, failure_rate: f64) -> Vec<Log> {
        logs.into_iter().filter(|_| rand::random::<f64>() > failure_rate).collect()
    }

Also would love stylistic guidance if you want me to go forward with implementing this.

Modified Files: