opensearch-project / opensearch-py

Python Client for OpenSearch
https://opensearch.org/docs/latest/clients/python/
Apache License 2.0
350 stars 175 forks source link

Avoid decoding data in request logging #485

Closed fabioasdias closed 11 months ago

fabioasdias commented 1 year ago

Hi all,

First, thanks for the library, really handy so I don't have to deal with sigv4 myself.

However, I'm running into an issue in this line:

https://github.com/opensearch-project/opensearch-py/blob/c8b04a5246dad6dcec05cf6f5bbf2eb14a9ec398/opensearchpy/connection/base.py#L232

where the request body is unnecessarily processed and ends up causing memory issues on a rather heavy bulk request. There's also the fact one might not want the body to be logged, there are security implications there in some cases. That's the reason behind the Pylint rule to use lazy evaluation on logging, one shouldn't pay the computational price for a log message that's not even persisted/desired.

wbeckler commented 1 year ago

Maybe something like if logger.isEnabledFor(debug) [decode the body]

dblock commented 1 year ago

100% we should not process the body unless explicitly needed for logging, please contribute a fix @fabioasdias?