API Watchdog

Usage

api-watchdog discover path/to/test/files

Will output the abbreviated result of the tests to stdout.

api-watchdog discover --email path/to/test/files

Will email the results of the tests to the relevant addresses (as determined by the email_to= field of the test. Test results are grouped so that only one email per address is sent per run.

Whjen using the CLI, these environment variables must be set to email results

• MAILGUN_API_URL
• MAILGUN_API_TOKEN
• MAILGUN_FROM

api-watchdog discover -o results_file.json path/to/test/files

Will serialize the WatchdogResultGroup object to the path specified by the -o flag.

Installation

pip install api-watchdog

Docker Installation

To run within Docker you can use the following command:

docker build -t api-watchdog .
docker run -e MAILGUN_API_URL=X -e MAILGUN_API_TOKEN=X -e MAILGUN_FROM=X api-watchdog api-watchdog discover path/to/test/files

To run tests within Docker you can use the following command:

docker build -t api-watchdog .
docker run api-watchdog python -m unittest discover tests

WatchdogTest format

The main way you'll interface with API Watchdog is through WatchdogTests. Each WatchdogTest has

• name (str): The name of the test
• target (url): The endpoint that the test targets
• method (str): HTTP request type, i.e. POST, GET
• expectations (Array[Expectation]): A list of requirements that the response must meet for the test to pass.
• payload (object): The json passed to the endpoint.

Expectation format

An Expectation describes where to find a piece of data in the response and what that piece of data should be in order for the test to pass. Each Expectation has

• selector (jq program): A string describing a jq program that selects the one or more elements to test against
• value (Any): a value to test equality against
• validation_type (ValidationType): An API Watchdog validation type used to validate the value/response. The value/response will be implicitly converted to this type. For example, if you specify 'float' and the value is an integer it will be implicitly converted to a float.
• level (Optional[ExpectationLevel]): How important an expectation is. Defaults to "critical"

ExpectationLevel

One of the strings:

• critical
• warning
• info Only "critical" expectations affect the success or failure of a WatchdogTest.

The selector format is a string that is a jq program. This allows for rich selection capabilities. For example:

    {
      "selector": ".message.knowledge_graph.nodes[\"MONDO:0005148\"].name",
      "value": "type 2 diabetes mellitus",
      "validation_type": "string"
    },

Is an Expectation that checks if a node is present in the knowledge graph of a TRAPI response.

The possible validation types are

• "string",
• "int",
• "float",
• "object",
• "bool",
• "null",
• "trapi.knowledge_graph",
• "trapi.node",
• "trapi.edge",
• "trapi.query_graph",
• "trapi.q_node",
• "trapi.q_edge",
• "trapi.attribute_constraint",
• "trapi.result",
• "trapi.node_binding",
• "trapi.edge_binding",
• "trapi.message",
• "trapi.query",
• "trapi.response",
• "trapi.async_query",
• "trapi.operation",
• "trapi.workflow",
• "trapi.attribute",
• "trapi.biolink_entity",
• "trapi.biolink_predicate",
• "trapi.curie",
• "trapi.log_entry",
• "trapi.log_level",
• "trapi.meta_edge",
• "trapi.meta_node",
• "trapi.meta_knowledge_graph",
• "trapi.meta_attribute"

What it is

An API monitoring utility that aspires to support:

• [ ] Validation
• [ ] Continuous Integration
• [ ] Multiple input and output formats
• [ ] Test discovery / minimal configuration

What it is not

• A way to regularly schedule tests against an endpoint, eg. cron, celery
• A way to specify api schemas eg. marshmallow, pydantic