Canonical transaction paths

netmilk commented 9 years ago

I propose to start using new syntax for addressing transactions compiled from the blueprint in hooks instead of transaction names. Transaction names will be supported for backward compatibility.

Transaction names were historically intended only as bread crubs — a human readable location of the specific transaction in the blueprint. Transaction name was never meant as a canonical transaction identifier and that's the reason for its terrible indeterminism.

Format will be a concatenation/serialization of the origin object:

Use colon : character as a delimiter
Colon character in API Name, Resource Name, Resource Group Name, Action Name or Example Name will be escaed with backslash
Examples are identified by string "Example " + its index in array starting from 1 (not 0)

Possible questions:

Omit leading "Example " string and use zero indexed position?
Use different delimiter?
Use delimiter string at the beginning of the path?
Does term "Canonical transaction path" make sense?
Examples

1. Full notation with multiple request-response pairs

# Some API Name

## Group Some Group Name

### Some Resource Name [/resource]

#### Some Action [GET]

+ Request (application/json)
+ Response 200(application/json)

+ Request (application/xml)
+ Response 200 (application/xml)

Transaction origin object:

{
  "apiName": "Some API Name",
  "resourceGroupName": "Some Group Name",
  "resourceName": "Some Resource Name",
  "actionName": "Some Action Name",
  "exampleName": "Example 2"
}

Compiled canonical path:

Some API Name:Some Group Name:Some Resource Name:Some Action Name:Example 2

2. Full notation without group

# Some API Name

### Some Resource Name [/resource]

#### Some Action [GET]

+ Request (application/json)
+ Response 200 (application/json)

Transaction origin object:

{
  "apiName": "Some API Name",
  "resourceGroupName": "",
  "resourceName": "Some Resource Name",
  "actionName": "Some Action Name",
  "exampleName": "Example 1"
}

Compiled canonical path:

Some API Name::Some Resource Name:Some Action Name:Example 1

3. Full notation without group and API name

### Some Resource Name [/resource]

#### Some Action [GET]

+ Request (application/json)
+ Response 200 (application/json)

Transaction origin object:

{
  "apiName": "",
  "resourceGroupName": "",
  "resourceName": "Some Resource Name",
  "actionName": "Some Action Name",
  "exampleName": "Example 1"
}

Compiled canonical path:

::Some Resource Name:Some Action Name:Example 1

4. Simplified notation

# GET /message
+ Response 200 (text/plain)

      Hello World

Transaction origin object:

{
  "apiName": "",
  "resourceGroupName": "",
  "resourceName": "/message",
  "actionName": "GET",
  "exampleName": "Example 1"
}

Compiled canonical path:

::/message:GET:Example 1

netmilk commented 9 years ago

Implemented here https://github.com/apiaryio/blueprint-transactions/pull/1. To be implemented in Dredd's hook handler.

honzajavorek commented 8 years ago

I am proposing a slightly different approach.

Introduction

The proposal below attempts to address following issues:

Transaction Examples are hard to identify (just by enumeration: Example 1, Example 2) and because they're implicit, people do not know their concept well and confuse them with Requests. I removed them from the Transaction Path. (https://github.com/apiaryio/blueprint-transactions/pull/3)
Since Transaction Examples and Request-Response pairs are different things, the original specification above doesn't allow addressing complex combinations, such as multiple Requests with multiple Responses. I included identifiers for Requests and Responses as components of the Transaction Path. (https://github.com/apiaryio/dredd/issues/25, https://github.com/apiaryio/dredd/issues/78, https://github.com/apiaryio/dredd/issues/294)
Resulting Transaction Path should be much more descriptive and unique. It still won't catch all scenarios though. Example of an edge case:
```
+ Request
+ Response 200 (application/json)
  + Body ...
+ Response 200 (application/json)
  + Body ...
```
However, while such thing is valid API Blueprint syntax, it is ambiguous and should be treated as such. Transaction Path won't distinguish between the two Responses and Dredd will refuse to consume such blueprint. That would be considered as correct and expected behavior, because Dredd can't possibly guess what to test.

Transaction Path (string)

Canonical, deterministic way how to address a single HTTP Transaction (single Request-Response pair). It can serve as a unique identifier of the HTTP Transaction.

The Transaction Path string is a serialization of the pathOrigin object according to following rules:

Colon : character as a delimiter.
Colon character, which happens to be part of any component of the path, is escaped with backslash character \.
No other characters than colon : are escaped.

Components

If certain component isn't available in the source data, it is present in both the pathOrigin object and the Transaction Path as an empty string.
The requestName component is defined as name of the request followed by parentheses containing effective value of the Content-Type header. Any of those two parts can be omitted. If both parts are present, they're separated by a single space character.
The responseName component is defined as HTTP code of the response followed by parentheses containing effective value of the Content-Type header. The Content-Type part can be omitted. If both parts are present, they're separated by a single space character.

Examples

Full Notation with Multiple Request-Response Pairs

# Sample API Name

## Group Sample Group Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Request (application/json)
+ Response 200 (application/json)

+ Request (application/xml)
+ Response 200 (application/xml)

Transaction Path Origin:

{
  "apiName": "Sample API Name",
  "resourceGroupName": "Sample Group Name",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "(application/xml)",
  "responseName": "200 (application/xml)",
}

Transaction Path:

Sample API Name:Sample Group Name:Sample Resource Name:Sample Action Name:(application/xml):200 (application/xml)

Full Notation with Implicit Request

# Sample API Name

## Group Sample Group Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Response 200

Transaction Path Origin:

{
  "apiName": "Sample API Name",
  "resourceGroupName": "Sample Group Name",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "",
  "responseName": "200",
}

Transaction Path:

Sample API Name:Sample Group Name:Sample Resource Name:Sample Action Name::200

Full Notation with Multiple Requests within One Transaction Example

# Sample API Name

## Group Sample Group Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Response 200 (text/plain)

+ Request Sample Request Name (application/json)
+ Request Another Sample Request Name (application/json)
+ Request (application/json)
+ Response 200 (application/json)
+ Response 401 (application/json)

Transaction Path Origin:

{
  "apiName": "Sample API Name",
  "resourceGroupName": "Sample Group Name",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "Another Sample Request Name (application/json)",
  "responseName": "401 (application/json)",
}

Transaction Path:

Sample API Name:Sample Group Name:Sample Resource Name:Sample Action Name:Another Sample Request Name (application/json):401 (application/json)

Full Notation with Multiple Requests Each Having Multiple Responses

# Sample API Name

## Group Sample Group Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Response 200 (application/json)
+ Response 401 (application/json)
+ Response 500 (application/json)

+ Request (application/xml)
+ Response 200 (application/xml)
+ Response 500 (application/xml)

Transaction Path Origin:

{
  "apiName": "Sample API Name",
  "resourceGroupName": "Sample Group Name",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "",
  "responseName": "401 (application/json)",
}

Transaction Path:

Sample API Name:Sample Group Name:Sample Resource Name:Sample Action Name::200 (application/json)

Full Notation with Resolution of the Effective Content-Type Value

# Sample API Name

## Group Sample Group Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Request
    + Headers

            X-Request-ID: 30f14c6c1fc85cba12bfd093aa8f90e3
            Content-Type: application/hal+json

+ Response 200

Transaction Path Origin:

{
  "apiName": "Sample API Name",
  "resourceGroupName": "Sample Group Name",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "(application/hal+json)",
  "responseName": "200",
}

Transaction Path:

Sample API Name:Sample Group Name:Sample Resource Name:Sample Action Name::200 (application/json)

Full Notation without Group

# Sample API Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Request (application/json)
+ Response 200 (application/json)

Transaction Path Origin:

{
  "apiName": "Sample API Name",
  "resourceGroupName": "",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "(application/json)",
  "responseName": "200 (application/json)"
}

Transaction Path:

Sample API Name::Sample Resource Name:Sample Action Name:(application/json):200 (application/json)

Full Notation without Group and API Name

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Request (application/json)
+ Response 200 (application/json)

Transaction Path Origin:

{
  "apiName": "",
  "resourceGroupName": "",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "(application/json)",
  "responseName": "200 (application/json)"
}

Transaction Path:

::Sample Resource Name:Sample Action Name:(application/json):200 (application/json)

Full Notation without Group and with API Name Containing a Colon

# My API: Revamp

### Sample Resource Name [/resource]

#### Sample Action Name [POST]

+ Request (application/json)
+ Response 200 (application/json)

Transaction Path Origin:

{
  "apiName": "My API: Revamp",
  "resourceGroupName": "",
  "resourceName": "Sample Resource Name",
  "actionName": "Sample Action Name",
  "requestName": "(application/json)",
  "responseName": "200 (application/json)"
}

Transaction Path:

My API\: Revamp::Sample Resource Name:Sample Action Name:(application/json):200 (application/json)

Simplified Notation

# GET /message
+ Response 200 (text/plain)

      Hello World

Transaction Path Origin:

{
  "apiName": "",
  "resourceGroupName": "",
  "resourceName": "/message",
  "actionName": "GET",
  "requestName": "",
  "responseName": "200 (text/plain)"
}

Transaction Path:

::/message:GET::200 (text/plain)

netmilk commented 8 years ago

Thanks @honzajavorek so much for this proposal incorporating solution for identifying multiple request and responses addressing #25 #294 #361 and others

My $0.50 would be:

Dredd should throw a description error and exit with non zero status code if some path isn't unique across all API documents loaded to the runtime
Please provide Transaction Path Origin and Path Examples for all Request-Response pairs in the "Full Notation with Multiple Requests Each Having Multiple Responses" example
When you are referring to some Transactionorigin object, please distinguish between the legacy Transaction Name Origin used for the Transaction Names or the actual Transaction Path Origin used for these Canonical Transaction Paths
Please stress the fact this change isn't braking the backward compatibility for integrations with the Transaction Name Origin object in the transaction object being passed to hook functions and hooks using Transaction Names.

The Transaction Name Origin and Transaction Name will be still supported but it will be considered as deprecated. The name compilation will be maintained in the reporters not in the transaction compiler in the future.

Regarding the Multiple Request and Responses support:

For keeping the backward compatibility I propose Dredd to test only the first Transaction (first request and first response in the Example) by default.
All other transactions from the Example will be passed to the runner as skipped (transaction.skip = true)
Any skipped transaction can be enabled by "unskipping" it in hooks (transaction.skip = false)
Introduce JS and CLI option for enabling all transactions (request-response pairs) to be tested by default e.g. --unskip-all or --all-pairs

kaikun213 commented 7 years ago

Is there a feature to see the transaction origins? E.g. as the transaction names with dredd --names a command dredd --origins. Otherwise I would suggest such feature.

honzajavorek commented 7 years ago

@kaikun213 There is no such feature right now and it wasn't planned. Could you share more about why it would be useful for you?

kaikun213 commented 7 years ago

@honzajavorek I am quite new to dredd and at the moment debugging my apib and hooks. Somehow some requests get not invoked and others get unwillingly invoked twice. So it would be useful to fastly list the origins and convert the used transaction names to origins.

honzajavorek commented 7 years ago

@kaikun213 I still don't quite understand how origins are more useful then the transaction names themselves. The transaction names are just origin object properties joined together. You can list origins and all the other info in the hooks just by inspecting the transaction object.

Introducing the transaction paths suggested in this issue would probably solve your problems though, because from what you write it seems to me you're experiencing some of the non-determinism of transaction names.

honzajavorek commented 6 years ago

Note: We should drop any structure from the transaction path. Its only purpose should be to "hash" individual requests present in the API description into a single human-readable declarative string, in a deterministic way. We should forget about XPath / JSON Pointer. This is mapping, not breadcrumbs.

For cases when this couldn't be deterministic, user should always be able to give the request/response a name in the API description to distinguish them and to address them.

halfvector commented 6 years ago

@honzajavorek Is there any movement on this? I'm very interested in clearly defining various request/response pairs for the same 200 status code for contract testing purposes.

honzajavorek commented 6 years ago

@halfvector We had some design sessions around this in April and per my last comment I have a better idea on how it should work. Currently there's no ETA though. It needs finalization of the design before anyone can implement it. I recognize it as one of the largest Dredd pain points, but as such it also requires me to carve out a significant block of uninterrupted time to tackle it.

halfvector commented 5 years ago

@honzajavorek Any updates on this? :)

honzajavorek commented 5 years ago

@halfvector Not really at this moment. We decided to focus on reviving and fixing the heart of Dredd's validation, Gavel. Let's see what we're up to after that.

apiaryio / dredd