PromQL results are irregular

[dtimm]

Metrics are not consistently available in log-cache when running the same query multiple times.

The first query below returns a data point, but subsequently it is only available intermittently:

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": [
      {
        "metric": {
          "deployment": "healthwatch-v1-3",
          "foundation": "acceptance-v1-3",
          "index": "b19ae1f4-0ed6-4f81-7a62-e4d6",
          "ip": "",
          "job": "healthwatch-forwarder",
          "origin": "healthwatch"
        },
        "points": [
          {
            "time": "1528909680000000000",
            "value": 0
          }
        ]
      }
    ]
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": [
      {
        "metric": {
          "deployment": "healthwatch-v1-3",
          "foundation": "acceptance-v1-3",
          "index": "b19ae1f4-0ed6-4f81-7a62-e4d6",
          "ip": "",
          "job": "healthwatch-forwarder",
          "origin": "healthwatch"
        },
        "points": [
          {
            "time": "1528909680000000000",
            "value": 0
          }
        ]
      }
    ]
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": [
      {
        "metric": {
          "deployment": "healthwatch-v1-3",
          "foundation": "acceptance-v1-3",
          "index": "b19ae1f4-0ed6-4f81-7a62-e4d6",
          "ip": "",
          "job": "healthwatch-forwarder",
          "origin": "healthwatch"
        },
        "points": [
          {
            "time": "1528909680000000000",
            "value": 0
          }
        ]
      }
    ]
  }
}

$ curl -G "https://log-cache.---.com/v1/promql" --data-urlencode 'query=redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="healthwatch-v1-3"}[4m]' -H "Authorization: $(cf oauth-token)" -k | jq .
{
  "matrix": {
    "series": []
  }
}

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/158336180

The labels on this github issue will be updated when the story is started.

[toddboom]

@dtimm Thanks for reporting this. I think it might be related to #57, but we should keep this open since you've got additional context here.

We're already digging into the issue and will update you once we've got it resolved.

[toddboom]

I think this was fixed when #57 was resolved, so I'm closing this out.