bug(`forge test`): JSON output should consistently adhere to verbosity flags

bowd commented 2 months ago

Component

Forge

Describe the feature you would like

It would be useful if the --json output of forge test would also contain the call traces and console.log output that the normal output contains.

Currently --json output's structure doesn't change if I increase verbosity, it looks like this:

{
  "test/oracles/ChainlinkRelayerFactory.t.sol:ChainlinkRelayerFactoryTest_deployRelayer": {
    "duration": "1ms 893us 708ns",
    "test_results": {
      "test_revertsWhenDeployingToAddressWithCode()": {
        "status": "Success",
        "reason": null,
        "counterexample": null,
        "logs": [
          { ... },
          { ... },
        ],
        "kind": {
          "Unit": {
            "gas": 104634
          }
        },
        "labeled_addresses": { ... },
        "duration": {
          "secs": 0,
          "nanos": 277375
        },
        "breakpoints": {}
      }
    },
    "warnings": []
  }
}

Additional context

I've been playing around with this idea of building an interactive testing environment for forge.

I find my self a lot of the time copy-pasting long test names in order to narrow down when I'm debugging or refactoring. I'd like a bit of UI that helps me by keeping track of failed tests and letting me dive in and out of the filtering options. I'd also like auto toggle verbosity on when I'm running a single test. And in the verbose mode have more control over the trace, collapsing longer sections, prettier structs, filtering some vm related logs, toggling setUp, toggling all calls to a certain function, etc. I think working with the traces is amazing but the UX can be improved a lot. I find this especially true when doing more complex fork test scenarios where there's a lot of stuff going on. And having a way to parse the forge test output with full verbosity would be amazing.

zerosnacks commented 2 months ago

Hi @bowd, regarding (1): I think --json should consistently adhere to verbosity rules. If you could point to cases where this is not the case this would be valuable. For (2): I would like to avoid adding additional tools due to the maintainance overhead but rather make modifications to existing tools to make sure they are user friendly. The best way for (2) is to split your proposal into individual tickets so they can be discussed in more detail. Generally avoid collapsing multiple diverging topics into a single ticket.

bowd commented 2 months ago

You're right, updated the issue description focusing on the --json output. Thanks!

ShantelPeters commented 2 months ago

Hi @bowd please can i be assigned to this issue

zerosnacks commented 1 month ago

foundry-rs / foundry