Centralised logging for API requests

traitecoevo / austraits-api-nectar

0 stars 1 forks source link

Centralised logging for API requests #2

Closed franksc closed 2 years ago

franksc commented 2 years ago

Incoming requests to the API service, received by the load balancer, should be centrally logged somewhere.

Individual instances running the Austraits API app just logging the requests they receive (forwarded from load balancer) isn’t great as you end up with request logs distributed among n instances/places, and these instances are notionally ephemeral anyway.

franksc commented 2 years ago

This has been implemented using:

Filebeat running on API instances, monitoring local nginx access log and pushing events to ...
Logtash running on a standalone instance, pushing log files to a Nectar Swift container using ...
logstash-output-swift plugin ...
With some patching for authentication to Nectar using OpenStack application credentials rather than username/passwords