Historically, it seems we sent to rsyslog so that datadog and logentries would both be able to process the logs. We are no longer using logentries, and the datadog direct ingested logs have a lot more metadata than the rsyslog forwarded one.
In this PR we:
Configure the docker containers to log with the json-file driver, and limit the size of each log file to 1MB.
We did not want to use the aws-logs driver, as we will be charged to store them in Cloudwatch, and we don't plan to use them there.
We could not configure it with the none driver, as that stopped all logging, even from the datadog agent ingesting directly.
Disable the functionality of the logentries plugin
Datadog configuration rollout
After this PR is merged, we do not need to update the datadog plugins districts to update the log functionality. The tasks will use the new configuration when they are deployed, as we are only changing how the plugins alter the task definition.
Logentries removal rollout
Logentries is already deactivated, but we still have the plugin installed on applications. This PR will remove the logentries configuration when an app is deployed. But we will still want to remove the plugin from all the districts using bcn plugin delete. Once it is removed from all districts, we can make a new PR to remove the code completely.
Note:
Due to the rsyslog parsing, using the service: tag facet was different for each container. But the datadog direct ingest maps service to the short image name, which containers share. For queries that need to filter on the specific container, we recommend using the container_name facet now. Here is an example of what to use in the future.
We have noticed duplicate logging to datadog for containers in ECS. Container logs are currently sent 2 ways.
Non-rsylog logs Rsyslog logs
Historically, it seems we sent to rsyslog so that datadog and logentries would both be able to process the logs. We are no longer using logentries, and the datadog direct ingested logs have a lot more metadata than the rsyslog forwarded one.
In this PR we:
json-filedriver, and limit the size of each log file to 1MB.aws-logsdriver, as we will be charged to store them in Cloudwatch, and we don't plan to use them there.nonedriver, as that stopped all logging, even from the datadog agent ingesting directly.Datadog configuration rollout
After this PR is merged, we do not need to update the datadog plugins districts to update the log functionality. The tasks will use the new configuration when they are deployed, as we are only changing how the plugins alter the task definition.
Logentries removal rollout
Logentries is already deactivated, but we still have the plugin installed on applications. This PR will remove the logentries configuration when an app is deployed. But we will still want to remove the plugin from all the districts using
bcn plugin delete. Once it is removed from all districts, we can make a new PR to remove the code completely.Note:
Due to the rsyslog parsing, using the
service:tag facet was different for each container. But the datadog direct ingest maps service to the short image name, which containers share. For queries that need to filter on the specific container, we recommend using thecontainer_namefacet now. Here is an example of what to use in the future.