Replace our external logging feature with Rsyslog

rooftopcellist commented 4 years ago

SUMMARY

This PR replaces https://github.com/ansible/awx/pull/5092 Issue: https://github.com/ansible/awx/issues/5155

This change adds rsyslog (https://github.com/rsyslog/rsyslog) as a new service that runs on every AWX node (managed by supervisord) in particular, this feature requires a recent version (v8.38+) of rsyslog that supports the omhttp module so as to send logs over HTTP/HTTPS (https://github.com/rsyslog/rsyslog-doc/pull/750)
The "external_logger" handler in AWX is now a SysLogHandler that ships logs to the local unix domain socket where rsyslog is configured to listen (by default, /var/run/tower/rsyslog/rsyslog.sock)
Every time a LOG_AGGREGATOR_* setting is changed, every AWX node reconfigures and restarts its local instance of rsyslog so that its fowarding settings match what has been configured in AWX.
unlike the prior implementation, if the external logging aggregator (splunk/logstash) goes temporarily offline, rsyslog will retain the messages and ship them when the log aggregator is back online.
4xx or 5xx level errors are recorded at /var/log/tower/rsyslog.err

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

API
Installer

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 5m 16s
awx-api : FAILURE in 9m 51s
awx-ui : SUCCESS in 6m 40s
awx-ui-next : SUCCESS in 10m 14s
awx-swagger : FAILURE in 5m 11s
awx-detect-schema-change : FAILURE in 3m 59s (non-voting)
awx-ansible-modules : SUCCESS in 5m 22s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 5m 38s
awx-api : FAILURE in 8m 05s
awx-ui : SUCCESS in 6m 08s
awx-ui-next : SUCCESS in 8m 56s
awx-swagger : FAILURE in 4m 30s
awx-detect-schema-change : FAILURE in 3m 28s (non-voting)
awx-ansible-modules : SUCCESS in 5m 05s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 5m 36s
awx-api : FAILURE in 12m 05s
awx-ui : SUCCESS in 10m 01s
awx-ui-next : SUCCESS in 9m 58s
awx-swagger : FAILURE in 4m 58s
awx-detect-schema-change : FAILURE in 4m 39s (non-voting)
awx-ansible-modules : SUCCESS in 7m 11s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 6m 41s
awx-api : FAILURE in 13m 00s
awx-ui : SUCCESS in 9m 14s
awx-ui-next : SUCCESS in 12m 06s
awx-swagger : FAILURE in 8m 19s
awx-detect-schema-change : FAILURE in 7m 05s (non-voting)
awx-ansible-modules : SUCCESS in 6m 59s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 6m 23s
awx-api : FAILURE in 12m 11s
awx-ui : SUCCESS in 10m 29s
awx-ui-next : SUCCESS in 11m 02s
awx-swagger : FAILURE in 6m 23s
awx-detect-schema-change : FAILURE in 5m 39s (non-voting)
awx-ansible-modules : SUCCESS in 6m 25s

kdelee commented 4 years ago

Connect the issue https://github.com/ansible/awx/pull/6108 also bug that should be resolved https://github.com/ansible/awx/issues/5822

jlmitch5 commented 4 years ago

UI work will involve:

Making it so that the Test button is disabled if the form contents are dirty (i.e. different than what is saved), as well as if the log aggregator enabled toggle is off
Adding tooltips for the disabled/enabled test button. These should be:

Test button enabled tooltip: "Send a test response to the configured log aggregator." Test button disabled tooltip: "Save and enable log aggregation before testing the log aggregator."

Updating the test POST to pass an empty object, as opposed to the current form values
Updating the language of the success toast message to: "Log aggregator test sent successfully."

Note the failure toast message/error modal logic does not need to be updated, what is there works.

jlmitch5 commented 4 years ago

UI work complete:

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 12m 26s
awx-api : FAILURE in 18m 16s
awx-ui : SUCCESS in 15m 30s
awx-ui-next : SUCCESS in 22m 00s
awx-swagger : FAILURE in 17m 21s
awx-detect-schema-change : FAILURE in 15m 41s (non-voting)
awx-ansible-modules : SUCCESS in 12m 44s

jakemcdermott commented 4 years ago

Screenshot from 2020-03-11 16-55-51 This is good.

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 7m 57s
awx-api : FAILURE in 17m 02s
awx-ui : FAILURE in 8m 11s
awx-ui-next : SUCCESS in 18m 11s
awx-swagger : FAILURE in 8m 05s
awx-detect-schema-change : FAILURE in 8m 11s (non-voting)
awx-ansible-modules : SUCCESS in 9m 19s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 4m 31s
awx-api : FAILURE in 16m 57s
awx-ui : FAILURE in 6m 23s
awx-ui-next : SUCCESS in 18m 16s
awx-swagger : FAILURE in 5m 58s
awx-detect-schema-change : FAILURE in 5m 18s (non-voting)
awx-ansible-modules : SUCCESS in 6m 59s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 5m 07s
awx-api : FAILURE in 11m 15s
awx-ui : FAILURE in 5m 09s
awx-ui-next : SUCCESS in 11m 31s
awx-swagger : FAILURE in 5m 33s
awx-detect-schema-change : FAILURE in 4m 50s (non-voting)
awx-ansible-modules : SUCCESS in 5m 44s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 5m 12s
awx-api : FAILURE in 10m 32s
awx-ui : FAILURE in 4m 35s
awx-ui-next : SUCCESS in 11m 07s
awx-swagger : FAILURE in 4m 29s
awx-detect-schema-change : FAILURE in 5m 00s (non-voting)
awx-ansible-modules : SUCCESS in 4m 51s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : MERGER_FAILURE
awx-api : FAILURE in 24m 14s
awx-ui : FAILURE in 14m 11s
awx-ui-next : SUCCESS in 24m 59s
awx-swagger : FAILURE in 13m 31s
awx-detect-schema-change : FAILURE in 14m 30s (non-voting)
awx-ansible-modules : SUCCESS in 13m 59s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 6m 32s
awx-api : FAILURE in 9m 42s
awx-ui : FAILURE in 5m 53s
awx-ui-next : SUCCESS in 9m 54s
awx-swagger : FAILURE in 5m 40s
awx-detect-schema-change : FAILURE in 6m 26s (non-voting)
awx-ansible-modules : SUCCESS in 7m 01s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 5m 51s
awx-api : SUCCESS in 12m 12s
awx-ui : FAILURE in 7m 39s
awx-ui-next : SUCCESS in 10m 44s
awx-swagger : SUCCESS in 12m 41s
awx-detect-schema-change : SUCCESS in 8m 32s (non-voting)
awx-ansible-modules : SUCCESS in 6m 58s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 6m 16s
awx-api : FAILURE in 11m 12s
awx-ui : SUCCESS in 6m 31s
awx-ui-next : SUCCESS in 10m 45s
awx-swagger : SUCCESS in 11m 38s
awx-detect-schema-change : SUCCESS in 11m 27s (non-voting)
awx-ansible-modules : SUCCESS in 5m 04s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 5m 35s
awx-api : SUCCESS in 14m 07s
awx-ui : SUCCESS in 7m 23s
awx-ui-next : SUCCESS in 11m 56s
awx-swagger : SUCCESS in 15m 48s
awx-detect-schema-change : SUCCESS in 13m 49s (non-voting)
awx-ansible-modules : SUCCESS in 6m 23s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 13m 10s
awx-api : FAILURE in 21m 42s
awx-ui : SUCCESS in 17m 28s
awx-ui-next : SUCCESS in 18m 55s
awx-swagger : SUCCESS in 21m 29s
awx-detect-schema-change : SUCCESS in 23m 20s (non-voting)
awx-ansible-modules : SUCCESS in 13m 05s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 5m 13s
awx-api : FAILURE in 11m 34s
awx-ui : SUCCESS in 12m 06s
awx-ui-next : SUCCESS in 15m 59s
awx-swagger : SUCCESS in 17m 08s
awx-detect-schema-change : SUCCESS in 11m 46s (non-voting)
awx-ansible-modules : SUCCESS in 6m 37s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 16m 48s
awx-api : FAILURE in 21m 46s
awx-ui : SUCCESS in 17m 48s
awx-ui-next : SUCCESS in 21m 10s
awx-swagger : SUCCESS in 21m 11s
awx-detect-schema-change : SUCCESS in 22m 01s (non-voting)
awx-ansible-modules : SUCCESS in 13m 33s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 12m 01s
awx-api : SUCCESS in 16m 34s
awx-ui : SUCCESS in 7m 49s
awx-ui-next : SUCCESS in 13m 14s
awx-swagger : SUCCESS in 16m 13s
awx-detect-schema-change : SUCCESS in 15m 09s (non-voting)
awx-ansible-modules : SUCCESS in 12m 18s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 12m 00s
awx-api : SUCCESS in 18m 57s
awx-ui : SUCCESS in 13m 54s
awx-ui-next : SUCCESS in 16m 52s
awx-swagger : SUCCESS in 18m 45s
awx-detect-schema-change : FAILURE in 19m 53s (non-voting)
awx-ansible-modules : SUCCESS in 7m 04s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 17m 25s
awx-api : SUCCESS in 21m 20s
awx-ui : SUCCESS in 18m 19s
awx-ui-next : SUCCESS in 23m 11s
awx-swagger : SUCCESS in 23m 35s
awx-detect-schema-change : FAILURE in 22m 40s (non-voting)
awx-ansible-modules : SUCCESS in 17m 45s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 5m 19s
awx-api : FAILURE in 10m 53s
awx-ui : SUCCESS in 9m 28s
awx-ui-next : SUCCESS in 11m 09s
awx-swagger : SUCCESS in 12m 40s
awx-detect-schema-change : FAILURE in 11m 20s (non-voting)
awx-ansible-modules : SUCCESS in 7m 08s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 6m 50s
awx-api : FAILURE in 18m 35s
awx-ui : SUCCESS in 8m 20s
awx-ui-next : SUCCESS in 10m 55s
awx-swagger : SUCCESS in 12m 23s
awx-detect-schema-change : FAILURE in 12m 38s (non-voting)
awx-ansible-modules : SUCCESS in 6m 12s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 7m 42s
awx-api : FAILURE in 10m 07s
awx-ui : SUCCESS in 7m 24s
awx-ui-next : SUCCESS in 15m 39s
awx-swagger : SUCCESS in 15m 46s
awx-detect-schema-change : SUCCESS in 15m 07s (non-voting)
awx-ansible-modules : SUCCESS in 5m 00s

softwarefactory-project-zuul[bot] commented 4 years ago

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 18m 01s
awx-api : FAILURE in 22m 08s
awx-ui : SUCCESS in 18m 24s
awx-ui-next : SUCCESS in 22m 34s
awx-swagger : FAILURE in 24m 33s
awx-detect-schema-change : FAILURE in 23m 44s (non-voting)
awx-ansible-modules : SUCCESS in 17m 46s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 6m 41s
awx-api : FAILURE in 13m 47s
awx-ui : SUCCESS in 12m 24s
awx-ui-next : SUCCESS in 11m 57s
awx-swagger : FAILURE in 14m 01s
awx-detect-schema-change : FAILURE in 13m 21s (non-voting)
awx-ansible-modules : SUCCESS in 7m 54s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 17m 09s
awx-api : FAILURE in 22m 17s
awx-ui : SUCCESS in 15m 30s
awx-ui-next : SUCCESS in 19m 10s
awx-swagger : FAILURE in 21m 03s
awx-detect-schema-change : FAILURE in 21m 08s (non-voting)
awx-ansible-modules : SUCCESS in 14m 04s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 6m 43s
awx-api : FAILURE in 13m 29s
awx-ui : SUCCESS in 10m 26s
awx-ui-next : SUCCESS in 14m 45s
awx-swagger : FAILURE in 15m 34s
awx-detect-schema-change : FAILURE in 14m 39s (non-voting)
awx-ansible-modules : SUCCESS in 6m 05s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : SUCCESS in 5m 55s
awx-api : FAILURE in 11m 39s
awx-ui : SUCCESS in 10m 56s
awx-ui-next : SUCCESS in 12m 13s
awx-swagger : SUCCESS in 13m 25s
awx-detect-schema-change : SUCCESS in 13m 14s (non-voting)
awx-ansible-modules : FAILURE in 4m 48s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 5m 34s
awx-api : SUCCESS in 11m 55s
awx-ui : SUCCESS in 7m 55s
awx-ui-next : SUCCESS in 12m 23s
awx-swagger : SUCCESS in 11m 50s
awx-detect-schema-change : SUCCESS in 11m 54s (non-voting)
awx-ansible-modules : SUCCESS in 6m 04s

ryanpetrello commented 4 years ago

recheck

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 25s
awx-api : SUCCESS in 9m 45s
awx-ui : SUCCESS in 6m 15s
awx-ui-next : SUCCESS in 11m 02s
awx-swagger : SUCCESS in 12m 45s
awx-detect-schema-change : SUCCESS in 11m 16s (non-voting)
awx-ansible-modules : SUCCESS in 4m 52s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 5m 32s
awx-api : SUCCESS in 14m 23s
awx-ui : SUCCESS in 6m 46s
awx-ui-next : SUCCESS in 16m 04s
awx-swagger : SUCCESS in 10m 56s
awx-detect-schema-change : SUCCESS in 15m 08s (non-voting)
awx-ansible-modules : SUCCESS in 5m 09s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 09s
awx-api : SUCCESS in 10m 50s
awx-ui : SUCCESS in 7m 09s
awx-ui-next : SUCCESS in 11m 43s
awx-swagger : SUCCESS in 10m 47s
awx-detect-schema-change : SUCCESS in 12m 21s (non-voting)
awx-ansible-modules : SUCCESS in 4m 43s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 18s
awx-api : SUCCESS in 10m 46s
awx-ui : SUCCESS in 6m 48s
awx-ui-next : SUCCESS in 10m 29s
awx-swagger : SUCCESS in 12m 25s
awx-detect-schema-change : FAILURE in 10m 30s (non-voting)
awx-ansible-modules : SUCCESS in 4m 36s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 14s
awx-api : SUCCESS in 9m 16s
awx-ui : SUCCESS in 6m 07s
awx-ui-next : SUCCESS in 11m 29s
awx-swagger : SUCCESS in 12m 49s
awx-detect-schema-change : FAILURE in 10m 46s (non-voting)
awx-ansible-modules : SUCCESS in 4m 46s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 20m 42s
awx-api : SUCCESS in 12m 37s
awx-ui : SUCCESS in 9m 56s
awx-ui-next : SUCCESS in 13m 47s
awx-swagger : SUCCESS in 12m 01s
awx-detect-schema-change : FAILURE in 15m 21s (non-voting)
awx-ansible-modules : SUCCESS in 7m 01s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 7m 05s
awx-api : SUCCESS in 12m 04s
awx-ui : SUCCESS in 10m 33s
awx-ui-next : SUCCESS in 12m 43s
awx-swagger : SUCCESS in 8m 10s
awx-detect-schema-change : FAILURE in 14m 17s (non-voting)
awx-ansible-modules : SUCCESS in 7m 14s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 6m 50s
awx-api : SUCCESS in 12m 32s
awx-ui : SUCCESS in 7m 35s
awx-ui-next : SUCCESS in 14m 29s
awx-swagger : SUCCESS in 13m 16s
awx-detect-schema-change : FAILURE in 15m 44s (non-voting)
awx-ansible-modules : SUCCESS in 7m 49s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 08s
awx-api : SUCCESS in 11m 15s
awx-ui : SUCCESS in 7m 29s
awx-ui-next : SUCCESS in 10m 36s
awx-swagger : SUCCESS in 10m 54s
awx-detect-schema-change : FAILURE in 10m 56s (non-voting)
awx-ansible-modules : SUCCESS in 4m 54s

softwarefactory-project-zuul[bot] commented 4 years ago

Build failed.

awx-api-lint : FAILURE in 7m 52s
awx-api : SUCCESS in 11m 12s
awx-ui : SUCCESS in 8m 54s
awx-ui-next : SUCCESS in 14m 56s
awx-swagger : SUCCESS in 12m 47s
awx-detect-schema-change : FAILURE in 12m 34s (non-voting)
awx-ansible-modules : SUCCESS in 5m 22s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 57s
awx-api : SUCCESS in 14m 03s
awx-ui : SUCCESS in 8m 41s
awx-ui-next : SUCCESS in 15m 09s
awx-swagger : SUCCESS in 13m 11s
awx-detect-schema-change : FAILURE in 11m 45s (non-voting)
awx-ansible-modules : SUCCESS in 6m 39s

shanemcd commented 4 years ago

LGTM, but let's make sure to follow up on the small things caught by @chrismeyersfsu and @wenottingham.

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 5m 47s
awx-api : SUCCESS in 17m 45s
awx-ui : SUCCESS in 7m 18s
awx-ui-next : SUCCESS in 23m 04s
awx-swagger : SUCCESS in 11m 39s
awx-detect-schema-change : FAILURE in 12m 43s (non-voting)
awx-ansible-modules : SUCCESS in 5m 33s

softwarefactory-project-zuul[bot] commented 4 years ago

Build succeeded.

awx-api-lint : SUCCESS in 4m 09s
awx-api : SUCCESS in 11m 03s
awx-ui : SUCCESS in 6m 30s
awx-ui-next : SUCCESS in 10m 57s
awx-swagger : SUCCESS in 12m 13s
awx-detect-schema-change : FAILURE in 10m 33s (non-voting)
awx-ansible-modules : SUCCESS in 5m 47s

ansible / awx