search

Elfocrash / Cosmonaut

🌐 A supercharged Azure CosmosDB .NET SDK with ORM support
https://cosmonaut.readthedocs.io
MIT License
341 stars 44 forks source link

RequestRateTooLarge in AppInsights #112

Open Mortana89 opened 5 years ago

Mortana89 commented 5 years ago
We're doing some load testing on our staging environment, and I'm seeing alot of the following in AI: Microsoft.Azure.Documents.RequestRateTooLargeException Cosmonaut.Diagnostics.CosmosEventCall+d__8`1.MoveNext  

Shouldn't these type of errors be silent?

Elfocrash commented 5 years ago

Hello @Mortana89,

No I don't think that these errors should be log-level silent (even though they are app level silent) because it is very handy to aggregate them and see how many of your requests failed in order to generate some interesting metrics, autoscaling azure functions etc. You can also base alerts on them to indicate that you need to take immediate action in terms of scaling. I think it's a handy thing to log.

Mortana89 commented 5 years ago

Hmm, I understand where you're going to, but the fact it's being logged under 'Exceptions' in AI is a bit annoying. I expect (near to) zero exceptions in this view, because else something has a bug in it. When the system is under load, the fact that we receive 'RequestRateTooLarge' is totally normal behavior, which clutters the AI views now. It would be better (I think) if this could be moved to a dependency 'failure' then as it's not a real bug, merely a behavior of the system (Cosmos DB) under pressure. What do you think?

Elfocrash commented 5 years ago

Exceptions are not meant to indicate bugs but rather exceptional behavior and I think this specific one falls under this category. I could potentially add a flag on the AI package level to eagerly ignore it but I know for a fact that many users base alerts and autoscaling on it (including myself) and I can't just remove it.

Mortana89 commented 5 years ago

Why not use the Cosmos DB monitoring for that? That's what we do, we send alerts when the cosmos db throttled requests are abnormally high. A flag would be useful, would that be much work?

Elfocrash commented 5 years ago

Nah it shouldn't be much work. The reason for the code level logging is consistency of logs. It leads to simpler and more consistent queries.