Improve logging for external resources

[davidmegginson]

Improve the logging for the external resources processed in the hxl.input module:

• pass a JSON object up and down the functions in the module to accumulate information, starting with either data() or make_input()
• catch exceptions, add the JSON info, and re-raise
• log an info message at the end of successfully opening the resource with the info in the JSON object

The goal is for the HXL Proxy to be able to capture this information and log it more granularly. Information to capture includes the following:

• external URL
• status code returned by external (if any)
• MIME type returned by external server (if any)
• content size returned by external server (if any)
• detected file extension (if any)
• computed data type (may differ from MIME type)
• computed data size (Excel files only)
• computed download time (Excel files only)

The HXL Proxy application layer can add the IP address making the request and other server-layer information.

[davidmegginson]

Info on how to do this: https://stackoverflow.com/questions/11232230/logging-to-two-files-with-different-settings

[davidmegginson]

Added second logger "hxl.REMOTE_ACCESS" with basic messages about remote URL access. Will add the Javascript stuff in a future release if needed, but this provides the basics of what the HXL Proxy is trying to open and what responses it gets.

[emmajane]

@davidmegginson Is there a PR that the ops team can review to make sure it will be sufficient? I'm not sure what the "javascript stuff" is that you're referring to so I'm not sure how to provide feedback on this...

[davidmegginson]

Apologies, I meant JSON. Collecting multiple points of information into a json structure is a big piece of work, that @danmihaila will have to triage; I put this small fix in place so that we will at least have the ability to see in the logs what external URLs are being accessed and what status codes are being returned.