Make connections between theory and json

[codefromthecrypt]

I've found it useful to compare an image to json, when explaining how things work. If we had a doc page that links in practical terms an image to json, to keys in that json, I think we'd be in a better place.

Ex.

is from this json

[
  [
    {
      "traceId": "bd7a977555f6b982",
      "name": "get",
      "id": "bd7a977555f6b982",
      "timestamp": 1458702548467000,
      "duration": 386000,
      "annotations": [
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548467000,
          "value": "sr"
        },
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548853000,
          "value": "ss"
        }
      ],
      "binaryAnnotations": []
    },
    {
      "traceId": "bd7a977555f6b982",
      "name": "get-traces",
      "id": "ebf33e1a81dc6f71",
      "parentId": "bd7a977555f6b982",
      "timestamp": 1458702548478000,
      "duration": 354374,
      "annotations": [],
      "binaryAnnotations": [
        {
          "key": "lc",
          "value": "JDBCSpanStore",
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          }
        },
        {
          "key": "request",
          "value": "QueryRequest{serviceName=zipkin-query, spanName=null, annotations=[], binaryAnnotations={}, minDuration=null, maxDuration=null, endTs=1458702548478, lookback=86400000, limit=1}",
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          }
        }
      ]
    },
    {
      "traceId": "bd7a977555f6b982",
      "name": "query",
      "id": "be2d01e33cc78d97",
      "parentId": "ebf33e1a81dc6f71",
      "timestamp": 1458702548786000,
      "duration": 13000,
      "annotations": [
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548786000,
          "value": "cs"
        },
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548799000,
          "value": "cr"
        }
      ],
      "binaryAnnotations": [
        {
          "key": "jdbc.query",
          "value": "select distinct `zipkin_spans`.`trace_id` from `zipkin_spans` join `zipkin_annotations` on (`zipkin_spans`.`trace_id` = `zipkin_annotations`.`trace_id` and `zipkin_spans`.`id` = `zipkin_annotations`.`span_id`) where (`zipkin_annotations`.`endpoint_service_name` = ? and `zipkin_spans`.`start_ts` between ? and ?) order by `zipkin_spans`.`start_ts` desc limit ?",
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          }
        },
        {
          "key": "sa",
          "value": true,
          "endpoint": {
            "serviceName": "spanstore-jdbc",
            "ipv4": "127.0.0.1",
            "port": 3306
          }
        }
      ]
    },
    {
      "traceId": "bd7a977555f6b982",
      "name": "query",
      "id": "13038c5fee5a2f2e",
      "parentId": "ebf33e1a81dc6f71",
      "timestamp": 1458702548817000,
      "duration": 1000,
      "annotations": [
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548817000,
          "value": "cs"
        },
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548818000,
          "value": "cr"
        }
      ],
      "binaryAnnotations": [
        {
          "key": "jdbc.query",
          "value": "select `zipkin_spans`.`trace_id`, `zipkin_spans`.`id`, `zipkin_spans`.`name`, `zipkin_spans`.`parent_id`, `zipkin_spans`.`debug`, `zipkin_spans`.`start_ts`, `zipkin_spans`.`duration` from `zipkin_spans` where `zipkin_spans`.`trace_id` in (?)",
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          }
        },
        {
          "key": "sa",
          "value": true,
          "endpoint": {
            "serviceName": "spanstore-jdbc",
            "ipv4": "127.0.0.1",
            "port": 3306
          }
        }
      ]
    },
    {
      "traceId": "bd7a977555f6b982",
      "name": "query",
      "id": "37ee55f3d3a94336",
      "parentId": "ebf33e1a81dc6f71",
      "timestamp": 1458702548827000,
      "duration": 2000,
      "annotations": [
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548827000,
          "value": "cs"
        },
        {
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          },
          "timestamp": 1458702548829000,
          "value": "cr"
        }
      ],
      "binaryAnnotations": [
        {
          "key": "jdbc.query",
          "value": "select `zipkin_annotations`.`trace_id`, `zipkin_annotations`.`span_id`, `zipkin_annotations`.`a_key`, `zipkin_annotations`.`a_value`, `zipkin_annotations`.`a_type`, `zipkin_annotations`.`a_timestamp`, `zipkin_annotations`.`endpoint_ipv4`, `zipkin_annotations`.`endpoint_port`, `zipkin_annotations`.`endpoint_service_name` from `zipkin_annotations` where `zipkin_annotations`.`trace_id` in (?) order by `zipkin_annotations`.`a_timestamp` asc, `zipkin_annotations`.`a_key` asc",
          "endpoint": {
            "serviceName": "zipkin-query",
            "ipv4": "192.168.1.2",
            "port": 9411
          }
        },
        {
          "key": "sa",
          "value": true,
          "endpoint": {
            "serviceName": "spanstore-jdbc",
            "ipv4": "127.0.0.1",
            "port": 3306
          }
        }
      ]
    }
  ]
]

And that json includes keys defined in a couple constants files like.. https://github.com/openzipkin/zipkin-java/blob/master/zipkin/src/main/java/zipkin/Constants.java https://github.com/openzipkin/zipkin-java/blob/master/zipkin/src/main/java/zipkin/TraceKeys.java

And that json was produced by using the zipkin api, documented here: http://zipkin.io/zipkin-api/#/

using commands like this:

# in one tab, start a server
$ MYSQL_USER=root STORAGE_TYPE=mysql ./mvnw -pl zipkin-server spring-boot:run
# in another curl it once to make a trace
$ curl -s 'localhost:9411/api/v1/traces?serviceName=zipkin-query&limit=1' > /dev/null
# and again to read a self-trace of above
$ curl -s 'localhost:9411/api/v1/traces?serviceName=zipkin-query&limit=1' |jq .

[codefromthecrypt]

key insights for the above is that inbound and outbound requests are in different spans, and that spans that include "cs" can log an "sa" annotation of where they are going. This helps when the destination protocol isn't zipkin instrumented, such as mysql's protocol.

[codefromthecrypt]

copy/pasting a description of endpoint from brave.

/**
 * Indicates the network context of a service recording an annotation with two
 * exceptions.
 *
 * When a BinaryAnnotation, and key is CLIENT_ADDR or SERVER_ADDR,
 * the endpoint indicates the source or destination of an RPC. This exception
 * allows zipkin to display network context of uninstrumented services, or
 * clients such as web browsers.
 */

https://github.com/openzipkin/brave/blob/fd3cdb921964a0d7164c29141fe6aac8099ae4cf/brave-core/src/main/java/com/twitter/zipkin/gen/Endpoint.java

[codefromthecrypt]

from gitter

@gena01 found this helpful https://youtu.be/EP4prWJIWvc?t=364

basically, it starts describing a graph with the "cs" "cr" etc relationships.

https://speakerdeck.com/sprsquish/zipkin-number-sfscala-meetup

In whatever section ends up holding the how-to on custom instrumentation, we should have a diagram so people can understand client vs server annotations. We'd be a-miss if we didn't make the diagrams indicate the reporting service because this is an extremely common concern.

For example, today @chemicL asked about what the endpoint should be for zipkin annotations. The reason this question comes up is that a lot of old instrumentation got this wrong. If our diagram can re-enforce with labels, we can save time and frustration.

For example, if a service is receiving a request ("sr"), the endpoint is the server doing the receiving. If a client is sending a request ("cs"), the endpoint is the client doing the sending. Finagle correctly refers to this as the "local endpoint", but the problem is that often people misconfigure it to be the remote one. We should try hard to make it clear that this is the thing logging (as the above brave docs say).

This particular issue will go away once we have a new model as this whole dillema is caused by it being possible to have multiple endpoints in the same span. For example, if https://github.com/openzipkin/zipkin/issues/939 is implemented, we would have no need for "cs" "cr" "sr" "ss", so no opportunity to misconfigure an endpoint for them.

[codefromthecrypt]

Here's an example used in sleuth. They put the annotations in the front page and also a diagram.

https://github.com/spring-cloud/spring-cloud-sleuth

The reason the diagram is there is that it is easier to figure out what things are when you look at one. Before we copy/paste that here :) we should know that tracers will make traces with different shapes. For example, In the above link, sleuth eagerly makes a local span after a server receives a request. In zipkin's homepage, there's a local span before the client send. Choices like these will happen and they aren't right or wrong.

Whatever we make here will show just a basic example, libraries still need to document themselves and have maintenance.