add feature to save rsyslogd queue content when external logging is misconfigured

[rchincholkar]

ISSUE TYPE

• Feature Idea

SUMMARY

rsyslog does not save the queue content when external logging is misconfigured. In case of a 404 error on the external logging service, a buffer will not be kept and any log events will be lost. For example, one time 404 error from an external logging server, we lost log events until the external logging service is fixed.

Additionally, rsyslogd does not save the queue content on service shutdown. can we add such feature to store the queue on rsyslogd shutdown or restart.

[ryanpetrello]

Hey @rchincholkar,

By default, we configure rsyslogd to spool to a disk-backed queue if the configured log aggregator (e.g., Splunk, ELK) cannot be reached.

https://github.com/ansible/awx/blob/devel/awx/main/utils/external_logging.py#L33

main_queue(queue.spoolDirectory="{spool_directory}" queue.maxdiskspace="{max_disk_space}g" queue.type="Disk" queue.filename="awx-external-logger-backlog")

Is this not working?

If rsyslogd itself isn't running for some reason, we're unlikely to add any additional fallback mechanism for spooling those messages somewhere.

[rchincholkar]

yeah, it is not working if we get any 404 error from the external log aggregator, even if the rsyslogd is running.

I came across rsyslog options like $MainMsgQueueSaveOnShutdown on and $ActionQueueSaveOnShutdown on that could save queue, but never tested them. may be a good idea to add to rsyslog config, if this works.

[ryanpetrello]

If there's some configuration option you can find that provides what you want, it's fairly simple to modify what AWX writes out (the code for omhttp config lives here: https://github.com/ansible/awx/blob/devel/awx/main/utils/external_logging.py#L47-L105)

If you discover something, pull requests are welcome 😄

[blomquisg]

@tchellomello You wanna take a crack at this one? https://github.com/ansible/awx/issues/9407#issuecomment-795981741