Too extensive logging

[mikahanninen]

Can you change log_response() and log_request() to log at DEBUG level instead of INFO level.

Especially response.body can contain a LOT of data which will unnecessarily fill the log.

[lucagiove]

I see the point but there is already a truncate mechanism to limit log at max 10.000 characters.

[mikahanninen]

Yeah but 10000 characters is quite a lot :) Instead of automatically logging this, this could be left at the user's responsibility and decision if they want to do the logging. Especially the response logging.

[robinmatz]

Personally, I like the feature of having the response logged at level info.

The idea to keep this by setting log level to debug would lead to a massive increase in overall log size, because you would now be logging at debug level for ALL libraries used in your automation project.

Maybe settle for a compromise of setting the the truncate char limit at library import?

[lucagiove]

I think for the moment I'll keep as it is. If there will be still issues with performance I might add something to not log body depending on content-length size.

[Eli-Black-Work]

@lucagiove Can this be reopened? 🙂 We're running into a similar issue, but for file uploads.

When we do a PUT/POST with form-data to upload a file, we end up with logs that look like this:

PUT Request : url=https://localhost/upload-file
 path_url=/upload-file
 headers={'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '2710108', 'Content-Type': 'multipart/form-data; boundary=41c0cb5af0asdfb2978c4717d615be771'} 
 body=b'--41c0cb5af0550asdf978c4717d615be771\r\nContent-Disposition: form-data; name="alwaysReplace"\r\n\r\ntrue\r\n--41c0cb5af0550asdf978c4717d615be771\r\nContent-Disposition: form-data; name="example"\r\n\r\example\r\n--41c0cb5af0550asdf978c4717d615be771\r\nContent-Disposition: form-data; name="filename"; filename="example.XML"\r\n\r\n\xff\xfe<\x00?\x00x\x00m\x00l\x00 \x00v\x00e\x00r\x00s\x00i\x00o\x00n\x00=\x00"\x001\x00.\x000\x00"\x00 \x00e\x00n\x00c\x00o\x00d\x00i\x00n\x00g\x00=\x00"\x00u\x00t\x00f\x00-\x001\x006\x00"\x00?\x00>\x00\r\x00\n\x00<\x00R\x00P\x00S\x00>\x00\r\x00\n\x00 \x00 \x00<\x00E\x00x\x00p\x00o\x00r\x00t\x00V\x00e\x00r\x00s\x00i\x00o\x00n\x00>\x006\x00.\x001\x000\x00.\x000\x000\x001\x00<\x00/\x00E\x00x\x00p\x00o\x00r\x00t\x00V\x00e\x00r\x00s\x00i\x00o\x00n\x00>\x00\r\x00\n\x00 \x00 \x00<\x00P\x00a\x00n\x00e\x00l\x00s\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00<\x00P\x00a\x00n\x00e\x00l\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00 \x00 \x00<\x00P\x00a\x00n\x00e\x00l\x00D\x00a\x00t\x00a\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00<\x00P\x00a\x00n\x00e\x00l\x00I\x00D\x00>\x00T\x00O\x00H\x00P\x00t\x00J\x00z\x00E\x00U\x00o\x00F\x00h\x00U\x009\x00B\x00C\x00q\x00Q\x00x\x00+\x00T\x00Q\x00=\x00=\x00<\x00/\x00P\x00a\x00n\x00e\x00l\x00I\x00D\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00<\x00P\x00a\x00n\x00e\x00l\x00T\x00y\x00p\x00e\x00>\x00h\x00x\x00i\x00l\x00A\x00Q\x00e\x00w\x00u\x00q\x003\x000\x00v\x00w\x004\x001\x00B\x003\x00V\x00W\x00n\x00A\x00=\x00=\x00<\x00/\x00P\x00a\x00n\x00e\x00l\x00T\x00y\x00p\x00e\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00<\x00N\x00a\x00m\x00e\x00>\x009\x00q\x005\x00E\x00o\x00d\x00d\x00u\x00W\x00L\x004\x00l\x00C\x004\x006\x00M\x00k\x00M\x00O\x003\x00x\x00w\x00=\x00=\x00<\x00/\x00N\x00a\x00m\x00e\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00<\x00D\x00e\x00s\x00c\x00r\x00i\x00p\x00t\x00i\x00o\x00n\x00>\x00N\x00/\x00G\x00Y\x00S\x00C\x005\x00V\x00O\x002\x00+\x00q\x007\x006\x00H\x00g\x00N\x00u\x002\x00V\x00z\x00Q\x00=\x00=\x00<\x00/\x00D\x00e\x00s\x00c\x00r\x00i\x00p\x00t\x00i\x00o\x00n\x00>\x00\r\x00\n\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 
[10KB more data here (10k characters == 10KB]

This inflates the size of our logs by a lot, since we're doing a lot of file uploads for our testing 😆

I'd like to propose this solution:

• Continue to truncate after 10k characters, by default.
• Add an option to RequestsLibrary.POST, RequestsLibrary.PUT, etc. to limit the size of the content body to a certain amount.
  • Alternatively, perhaps add support for passing a custom logging function to RequestsLibrary.POST, etc., so that the user could do whatever they like and then do the logging themselves.