Introduce a method to embed Test Cases in the specification

[jeluard]

It is desirable to have a way to define valid Test Cases in the specification.

This would allow to:

• fully clarify the specification and detail expected outcome
• improve implementations compatibility
• allow for automated tests of implementations

The following is a proposed syntax allowing this. It builds on top of current usage of example in the specification while being light and OpenAPI compliant.

Proposed syntax consist of:

• defining a top-level x-test-cases map property
• having matching (i.e. same id) examples per component (parameters, requestBody and response)

The combination of parameters, requestBody and associated path properties form the Request. The result of the execution of this Request by a node must match the associated Response definition.

x-test-cases is a Map[string, TestCase object] (defined as an OpenAPI extension) allowing to list high-level details of all Test Cases per path, with TestCase object defined as:

• description a high-level description of the TestCase
• expectedResponse

If a response is too big to be added inline, it can be defined as a URI via externalValue.

Here is an example:

post:
  tags:
    - Beacon
    - ValidatorRequiredApi
  summary: "Publish a signed block."
  operationId: "publishBlindedBlockV2"
  description: |
    Instructs the beacon node to use the components of the `SignedBlindedBeaconBlock` to construct and publish a 
    `SignedBeaconBlock` by swapping out the `transactions_root` for the corresponding full list of `transactions`.
    The beacon node should broadcast a newly constructed `SignedBeaconBlock` to the beacon network,
    to be included in the beacon chain. The beacon node is not required to validate the signed
    `BeaconBlock`, and a successful response (20X) only indicates that the broadcast has been
    successful. The beacon node is expected to integrate the new block into its state, and
    therefore validate the block internally, however blocks which fail the validation are still
    broadcast but a different status code is returned (202). Before Bellatrix, this endpoint will accept 
    a `SignedBeaconBlock`. The broadcast behaviour may be adjusted via the `broadcast_validation`
    query parameter.
+ x-test-cases:
+   ValidRequest:
+     description: "A valid request"
+     expectedResponse: "200"
+   InvalidRequest:
+     description: "A request missing proper signature"
+     expectedResponse: "400"
  parameters:
    - name: broadcast_validation
      in: query
      required: false
      description: |
        Level of validation that must be applied to a block before it is broadcast.

        Possible values:
        - **`gossip`** (default): lightweight gossip checks only
        - **`consensus`**: full consensus checks, including validation of all signatures and
          blocks fields _except_ for the execution payload transactions.
        - **`consensus_and_equivocation`**: the same as `consensus`, with an extra equivocation
          check immediately before the block is broadcast. If the block is found to be an
          equivocation it fails validation.

        If the block fails the requested level of a validation a 400 status MUST be returned
        immediately and the block MUST NOT be broadcast to the network.

        If validation succeeds, the block must still be fully verified before it is
        incorporated into the state and a 20x status is returned to the caller.
      schema:
        $ref: '../../../beacon-node-oapi.yaml#/components/schemas/BroadcastValidation'
+     examples:
+        ValidRequest:
+          value: gossip
+        InvalidRequest:
+          value: gossip
    - in: header
      schema:
        $ref: '../../../beacon-node-oapi.yaml#/components/schemas/ConsensusVersion'
      required: true
      name: Eth-Consensus-Version
      description: "Version of the block being submitted."
+     examples:
+        ValidRequest:
+          value: altair
+        InvalidRequest:
+          value: altair
  requestBody:
    description: "The `SignedBlindedBeaconBlock` object composed of `BlindedBeaconBlock` object (produced by beacon node) and validator signature."
    required: true
    content:
      application/json:
        schema:
          oneOf:
            - $ref: '../../../beacon-node-oapi.yaml#/components/schemas/SignedBeaconBlock'
            - $ref: "../../../beacon-node-oapi.yaml#/components/schemas/Altair.SignedBeaconBlock"
            - $ref: "../../../beacon-node-oapi.yaml#/components/schemas/Bellatrix.SignedBlindedBeaconBlock"
            - $ref: "../../../beacon-node-oapi.yaml#/components/schemas/Capella.SignedBlindedBeaconBlock"
            - $ref: "../../../beacon-node-oapi.yaml#/components/schemas/Deneb.SignedBlindedBeaconBlock"
+       examples:
+         ValidRequest:
+           value:
+             message: ...
+             signature: ...
+         InvalidRequest:
+           value:
+             message: ...
      application/octet-stream:
        schema:
          description: "SSZ serialized block bytes. Use content type header to indicate that SSZ data is contained in the request body."
  responses:
    "200":
      description: "The block was validated successfully and has been broadcast. It has also been integrated into the beacon node's database."
    "202":
      description: "The block failed validation, but was successfully broadcast anyway. It was not integrated into the beacon node's database."
    "400":
      description: "The `SignedBlindedBeaconBlock` object is invalid or broadcast validation failed"
      content:
        application/json:
          schema:
            $ref: "../../../beacon-node-oapi.yaml#/components/schemas/ErrorMessage"
-          example:
-            code: 400
-            message: "Invalid block: missing signature"
+         examples:
+           InvalidRequest:
+             value:
+               code: 400
+               message: "Invalid block: missing signature"
    "415":
      $ref: '../../../beacon-node-oapi.yaml#/components/responses/UnsupportedMediaType'
    "500":
      $ref: '../../../beacon-node-oapi.yaml#/components/responses/InternalError'
    "503":
      $ref: '../../../beacon-node-oapi.yaml#/components/responses/CurrentlySyncing'

Previous work

This proposition takes inspiration from the following works:

• https://docs.apimatic.io/specification-extensions/swagger-test-cases-extensions/
• https://github.com/apideck-libraries/portman

[dapplion]

I don't follow exactly what problem is this trying to address, could you expand more on the motivation?

[jeluard]

The high-level idea is to have comprehensive Test Cases that can be used with real chain data. e.g. right now there is no clear reproducible way to trigger specific errors reproducibly. The idea is similar to what can be found in execution-apis repository

Note that an alternative would be to have separate files defining individual Tests e.g. using markdown syntax