search

tomkerkhove / promitor

Bringing Azure Monitor metrics where you need them.
https://promitor.io
MIT License
252 stars 92 forks source link

Reduce telemetry noise #1625

Open tomkerkhove opened 3 years ago

tomkerkhove commented 3 years ago

Proposal

Reduce telemetry noise by:

Component

Resource Discovery, Scraper

Contact Details

No response

locmai commented 3 years ago

Hi again! We are having a logging noise issue as well with both scraper and resource discovery even when we set the log level to error.

[18:40:15 INF] Request finished in 0.5297ms 200 text/plain; version=0.0.4
[18:40:17 INF] Request starting HTTP/1.1 GET http://x.y.z.w/api/v1/health?includeDependencies=false

Is this the same issue?

And could it possibly because of this webHostBuilder with no log level configuration? https://github.com/tomkerkhove/promitor/blob/d5647fab740357ac57b617992fb34dc74d8be243/src/Promitor.Agents.Core/AgentProgram.cs#L33

tomkerkhove commented 3 years ago

I will add this to the checklist but the whole Serilog setup has to change for sure.

geertn commented 2 years ago

We have the same issue. Is there any known workaround? This is currently our biggest source of logging adding up to our costs.

│ [09:18:12 INF] Request starting HTTP/1.1 GET http://10.11.10.181:8888/api/v1/health?includeDependencies=false - -                                                                                           │
│ [09:18:12 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on  │
│ [09:18:12 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery) in 0.2218ms                                                            │
│ [09:18:12 WRN] {"RequestMethod": "GET", "RequestHost": "http://10.11.10.181:8888", "RequestUri": "/api/v1/health", "ResponseStatusCode": 200, "RequestDuration": "00:00:00.0004844", "RequestTime": "2022-0 │
│ [09:18:12 INF] Request finished HTTP/1.1 GET http://10.11.10.181:8888/api/v1/health?includeDependencies=false - - - 200 60 application/json;+charset=utf-8 0.6816ms                                         │
tomkerkhove commented 2 years ago

No but this will be fixed in the next version, sorry about that.

locmai commented 2 years ago

We have turned off the log on all environments for a while now to save the logging storage cost. We only turn it on QA env for debugging.

cktwo commented 2 years ago

Any news?

tomkerkhove commented 2 years ago

Nope but open to contributions for it though!