OpenBazaar / openbazaar-go

OpenBazaar 2.0 Server Daemon in Go
MIT License
993 stars 283 forks source link

Validate API responses in python QA tests #1789

Open drwasho opened 5 years ago

drwasho commented 5 years ago

Our python QA tests are excellent end-to-end test that replicates a variety of order-related scenarios to ensure that code changes do not break core protocol functionality. However, these tests do not tell us:

  1. If the responses are returned in the expected format
  2. If the values in a JSON response are of the correct type

This issue proposes that we start testing both! This will give us some confidence that the server is now significantly less likely to give us a false-positive for a given QA test.

As a simple example, for POST /ob/listing (i.e. creating a new listing), a typical response would be:

{
    "slug": "eth-physical-order-testing-w-options"
}

We should validate:

  1. The response is JSON
  2. There is only one key with a name of slug
  3. The value of this key is a string

A complex example would be POST /wallet/spend:

{
    "amount": "27404",
    "confirmedBalance": "1944250",
    "currency": {
        "code": "TLTC",
        "currencyType": "crypto",
        "divisibility": 8,
        "name": "Testnet Litecoin"
    },
    "memo": "Test spend testnet Litecoin",
    "orderId": "",
    "timestamp": "2019-10-11T03:02:40Z",
    "txid": "f9df310811c05bb8f21cae89cc5efc0c9bb18425ba16936267cd03fc09bb5822",
    "unconfirmedBalance": "0"
}

Here we should validate:

  1. The each key names and currency object
  2. The value types corresponding to each key (and nested object)
drwasho commented 5 years ago

The first batch of work is being done here: https://github.com/OpenBazaar/validation-obgo-tests

This issue is still open as these validations need to be integrated into the QA test suite. A way that might be done is to create 2 more folders in ./qa from the validation-obgo-tests repo:

  1. schema => contains the schema JSON
  2. responses => contains example JSON responses from the server, which we use to infer the schema from

Next, we'd validate the JSON shortly after lines like this: https://github.com/OpenBazaar/openbazaar-go/blob/master/qa/listings.py#L59