search

istio / old_issues_repo

Deprecated issue-tracking repo, please post new issues or feature requests to istio/istio instead.
37 stars 9 forks source link

Lots of metrics that I'm not injecting #290

Open jayunit100 opened 6 years ago

jayunit100 commented 6 years ago

Hi istio ! Have really enjoyed my initial POC w/ this stuff on GKE !!! Now that its time to get serious and try to create some finer grained information, and I ran into some issues, however.

I beleive this issue is a bug, because it makes it impossible, or extremely cumbersome, to use istio to investigate a single namespace of interest... but if not, I'm happy to close it .

It appears that the istio prometheus / grafana defaults

As an example, here are the metrics in my prometheus instance. many of which correspond to namespaces that were created many months before I put Istio on my cluster: (the hub45_istio metrics are valid, all other ones are unwanted entirely, and presumeably, shouldnt even be spotted by envoy as I never labelled those namespaces).

screen shot 2018-04-16 at 1 16 33 pm

1) Is this a bug at the envoy level ? I assume that Istio by default would only be reporting metrics for the namespaces I actually cared about / labelled.

2) Note that, in this cases, I've actually also deleted all pods in istio-system, just to see if that fixed things. It didnt seem to.

jayunit100 commented 6 years ago

1) Interesting. What I found in common was...... these are all HTTP services. So the missing services for envoy issue is a non-issue.

2) However, I'm still going to suggest that maybe the older services (created before I istio'd my cluster) being present is a probblem.

Starting to think maybe its just a config issue. Not sure though !

douglas-reid commented 6 years ago

@jayunit100:

RE: bytes-in/out for HTTP services

We are in-progress of adding total bytes sent/received for http services (current support is request.size and response.size, but this will add headers, etc. to the calculation).

RE: Old services

Are the service definitions for these services still in the API server?

jayunit100 commented 6 years ago

I see them for http services.

I don’t see them for https services.

jayunit100 commented 6 years ago

Thanks for the response btw, just curious if that HTTP above was a typo? Curious what challenges there are for https that raw bytes in and out are significantly different then http. I’m also trying to learn more about Istio internals so, could be my own ignorance.

douglas-reid commented 6 years ago

HTTPS (and other TCP protocols) should be there now (connection.sent.bytes_total and connection.received.bytes_total). The difference is that for HTTP, we had been relying on the request/response size attributes, which were including strictly reporting on the body. We are adding to HTTP support by adding attributes that include total size as well.

Does that help?

jayunit100 commented 6 years ago

http://35.184.76.105:3000/d/1/istio-dashboard?refresh=5s&orgId=1 , FYI, this is my istio dashboard. Anything look fishy? I only see metrics for HTTP based services, which definetly makes me think something is wrong w/ HTTPS based metrics?

douglas-reid commented 6 years ago

@jayunit100 one thing I notice is that your HTTP services are showing up as TCP services. I think this may be a configuration issue. Are the ports for those services labeled with the prefix of "http" ?