Logging

[silverdaz]

Correlation IDs are now included in the log record

Describe the pull request:

• [ ] Bug fix
• [ ] Functional change
• [ ] New feature
• [x] Code cleanup
• [x] Build system change
• [ ] Documentation change
• [ ] Language translation

Pull request long description:

The package __init__.py updates the logger class, and supplies one where we force correlation ids to be logged. utils/ogging.py contains the new logger class, and a utility to find the correlation id in the context of the caller. Unless the correlation id is itself specified in the log calls, we search up the call stack in the local variables of the caller and exits as soon as we find it, with a maximum of "levels" (initially fixed at 10, ie we search 10 callers' context up the stack). We return None otherwise.

Related issues:

Fixes #11

Additional information:

We swap the bootstrap default from s3 to posix. S3 was extremely slow to read (when about to decrypt). This is not the case for POSIX. We added an extra test for ingesting a 1GB file.

Mentions:

@viklund: You might like that solution. Code change in only 2 places.

[silverdaz]

With commit #d5cb7bb, I remove the need for searching the call stack. I essentially stash the correlation id in something that acts as a global variable across the package and all the loggers use that variable. The key place for setting the correlation id is when we pick a message from rabbitmq. Remember that a python process is single-threaded, there is no real parallelism.