Azure / azure-cosmosdb-java

Java Async SDK for SQL API of Azure Cosmos DB
MIT License
54 stars 61 forks source link

Direct TCP: Address back pressure issue that causes Out of Direct Memory Access Buffer Memory error #228

Closed David-Noble-at-work closed 5 years ago

David-Noble-at-work commented 5 years ago

Good news

Endpoints are resilient enough to recover from major errors like out of direct memory access buffers and closures on channel acquisition.

Bad news

Endpoints are prone to out of direct memory access buffers when request rates are high and its channels are on fast links.

Analysis

Out of direct memory access buffer errors are typically the result of back pressure. Back pressure is typically the result of sending/receiving more data than a client channel can consume and/or—to be determined—writing more data than the client channel’s server can consume. The result is that direct memory access buffers are consumed until they’re exhausted.

A potential resolution: slow down the server by refusing to read until the back pressure is relieved and/or slow down writes. At least one viable option is briefly discussed at:

and: