Draft: Implement DynamoDB storage component

devinsba commented 5 years ago

There's a couple things here that I need opinions on:

I've taken inspiration from haystack and put the full encoded span in the database. This allowed the span table to only have the attributes that are required by our queries in the table. Due to the fact that some of our queries (those without service or span) end up being table scans this will help us limit the response sizes.
I'm not super happy with the dependencies table implementation, I'd like that to be better if someone would take a look

devinsba commented 5 years ago

while letting this soak for a while, I've realized I have 1 too many tables since 2 of them have the same number and types of fields. I can consolidate that to save people some $ probably. There should be a changeset with that update this weekend

devinsba commented 5 years ago

Was discussing with Adrian the queries, I'm going to post a few here in comments

Get spans by name

{
  "TableName": "zipkin-spans",
  "IndexName": "span_name",
  "ScanIndexForward": false,
  "ProjectionExpression": "trace_id, trace_id_64, span_timestamp",
  "KeyConditionExpression": "span_name = :span_name AND span_timestamp_id BETWEEN :timestamp_id_lower_bound AND :timestamp_id_upper_bound",
  "ExpressionAttributeValues": {
    ":timestamp_id_lower_bound": {
      "N": "28653312812297787557491507200000"
    },
    ":span_name": {
      "S": "get"
    },
    ":timestamp_id_upper_bound": {
      "N": "28656500409692171312084461551615"
    }
  }
}

Get spans for service name and tag=value

{
  "TableName": "zipkin-spans",
  "IndexName": "local_service_name",
  "ScanIndexForward": false,
  "ProjectionExpression": "trace_id, trace_id_64, span_timestamp",
  "FilterExpression": "#tag_key0 = :tag_value0",
  "KeyConditionExpression": "local_service_name = :local_service_name AND span_timestamp_id BETWEEN :timestamp_id_lower_bound AND :timestamp_id_upper_bound",
  "ExpressionAttributeNames": {
    "#tag_key0": "tag.local"
  },
  "ExpressionAttributeValues": {
    ":timestamp_id_lower_bound": {
      "N": "28653312812297787557491507200000"
    },
    ":timestamp_id_upper_bound": {
      "N": "28656500409692171312084461551615"
    },
    ":tag_value0": {
      "S": "app"
    },
    ":local_service_name": {
      "S": "frontend"
    }
  }
}

Scan for spans between endTs and endTs-lookback

{
  "TableName": "zipkin-spans",
  "ProjectionExpression": "trace_id, trace_id_64, span_timestamp",
  "FilterExpression": "span_timestamp_id BETWEEN :timestamp_id_lower_bound AND :timestamp_id_upper_bound",
  "ExpressionAttributeValues": {
    ":timestamp_id_lower_bound": {
      "N": "28653312812297787557491507200000"
    },
    ":timestamp_id_upper_bound": {
      "N": "28656500409692171312084461551615"
    }
  }
}