[External logger] Add consistent metadata for tracking logs related to a job

[AlanCoding]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

• API

SUMMARY

Logs related to a particular job do not have consistent metadata that would allow grouping them in an external log aggregator

ENVIRONMENT

• AWX version: any
• AWX install method: openshift, minishift, docker on linux, docker for mac, boot2docker
• Ansible version: X.Y.Z
• Operating System:
• Web Browser:

STEPS TO REPRODUCE

Enable external logs for both awx logs & the job events logs

EXPECTED RESULTS

A way to filter logs in the external log aggregator such that it returns everything related to job with id=3244, for example.

ACTUAL RESULTS

The job events logger has a field job which points to the id of the related job, but there is no such thing in AWX logs, even which those logs are obviously exclusively about a particular job. We have many log statements, here is one:

https://github.com/ansible/awx/blob/4f5a0d5a08c6081389088f7aa7c9159bd4ab2a7f/awx/main/tasks.py#L999-L1000

The external log aggregator receives logs from all instances in the cluster, so someone would logically like to group logs related to a single job as it's path traversed multiple instances. One instance submits the job to the queue, and another runs it, for example.

ADDITIONAL INFORMATION

[zyphermonkey]

The splunk callback plugin uses a random uuid as a common field to link together all events from the same job.
Might it be possible to use something like that here?