Would it be possible to reuse loggregator for syslog?

[HDYA]

It would definitely be much more convenient if we could pass syslog through loggregator and reuse the nozzle.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/153987604

The labels on this github issue will be updated when the story is started.

[siennathesane]

@HDYA would you be able to clarify what your question is?

[hev]

@HDYA we would be interested in hearing more about this use case. It is one that we have considered doing and have some ways of accomplishing, but generally we have also seen a need for separate distinct streams.

[HDYA]

Sorry @mxplusb , It should be a suggest/feature request to enable syslog receiving for loggregator, then we could configure rsyslog to send syslogs to third-party log analytics services via loggregator and corresponding nozzle.

[siennathesane]

@HDYA there is a syslog release for BOSH which can be deployed to forward syslog messages anywhere you want. So I can understand the ask, what is the use case for syslog messages to go through the loggregator instead of being forwarded?

[hev]

I just wanted to follow up that we think this is a good candidate for running an isolation segment of Loggregator. We are in the process of evaluating that work now and will revisit this as a potential use case for issolation. It would still require some log forwarding from syslog-release into loggregator.

[HDYA]

Thanks for considering that!

I do believe that if both syslog and metrics share the same pattern of path to log analytics service, it would reduce complexity and save a lot of time during development and debug if there was a problem. Further if they can share exactly the same path, it would make HA and scaling simpler too (although it might cause more overhead and cost)

[NingKuang]

Hi, Adam, Any update or timeline on this request? We originally have an agent colleting both VM level metrics and syslog, now as Bosh HM system metrics is forwarded to loggregator, keeping this agent just for syslog does not make sense from engineering point of view. In addition, it is not a good customer experience as the customer has to install/maintain/coordinating two apps (the agent and the nozzle). It is definitely a great feature that will benefit the customers and for 3re party integrations.

[hev]

@NingKuang Thanks for providing these additional details. We are still collecting details about use case before getting started on this.

[gmmeyer]

I work on the Datadog Cloud Foundry integration. At some point we'll add our (still in beta) logs product to our Cloud Foundry integration. While collecting container logs ought to be pretty straightforward, collecting infra logs is going to be more of an issue.

To be honest, I thought infrastructure logs were routed through the loggregator. My planning all assumed this was the case. I didn't realize they weren't until recently. I think it would make sense to route them in this way. The loggregator adds a lot of context to the metrics that are valuable and harder to get on the agent. And, since Cloud Foundry also knows things about the services that the Datadog Agent does not and cannot, it would be more straightforward for it to collect logs.

From our perspective, it makes sense to route these logs through the loggregator. While we can grab these logs through our agent, it makes a lot more sense from our perspective to use the loggregator.

[NingKuang]

Hi, @ahevenor , Any update on this ? We have customer asking for this feature: getting syslog forwarded to log aggregator.

[hev]

@NingKuang no meaningful progress to share. Thanks for bumping this though. I hope to make progress on it soon. Keeping the issue open for continued visibility.

[Freakin]

I'm not sure if I like this approach without the isolation segment loggregator that @ahevenor mentioned. Syslog forwarder on local systems is quite reliable, and can be counted on even if there are loggregator problems.

[hev]

@Freakin good point. Thus far we haven't really found a need to address this. There are some correlation reasons that @gmmeyer mentioned that are a reason for this but this is at the expense for the potential for loss. Isolation segments can help isolate this loss but not eliminate this. This is in service of preventing back pressure (important for a multi-tenant platform) something components do not frequently encounter. At the moment it seems like this is most likely to stay as is for a while still.

That said, we are looking at starting to put on demand service instance logs into the firehose. Since these service instance are created by app developers, app developers should have access to thier logs.

[jtuchscherer]

Hi there,

we have a couple of updates about this issue:

1. The spike to investigate the forwarding of component syslog to the logging pipeline is coming up in our backlog: https://www.pivotaltracker.com/story/show/160326519

2. With our work on loggregator in isolation segments wrapping up, we have some work around: You can push a syslog drain app (for an example look here). This app has a syslog endpoint and will log all its messages to loggregator. You can use this app as a syslog endpoint for your component logs. This approach basically allows you to have component logs in the firehose. In order to not stress the normal loggregator system too much, I would consider using an logging system in an isolation segment. Also, cf service discovery will allow you to address the app on its internal domain name and you can spare the cf router from additional traffic

Having all of this said, we know many CF deployments where the additional traffic from component logs would push the loggregator pipeline to its limits. Operators will have to carefully evaluate whether or not they really want that additional stress on their logging pipeline. This stress would ultimately lead to a higher log message loss for the application logs.

Even if we were to add this feature, it would be on an opt-in basis and integrations like the Datadog CF integration could not rely on the component logs being available on all CF deployments.

Given all of this, I am not very optimistic that this feature will ever land in Cloud Foundry as a fully supported feature. However, we are still going to investigate this matter further when the story comes up in the backlog.

I am going to close this issue for now. Feel free to follow along the linked tracker story with our investigation and reopen this issue if you feel very stronlgy about this. You can also reach out to our slack channel

[NingKuang]

Thanks @jtuchscherer For enterprise customers who want an integrated logging solution (with both syslog and the CF system logs), do you recommend them to write their own syslog drain for scalability, instead of leverage loggregator? Thanks.