Request to expand DSC Log Resource to allow ship logs

[mikehumphrey]

The DSC Log Resource is a powerful tool in providing rich feedback from within the configuration process. However, this may not scale very well since the logs could potentially be spread across multiple DSC Servers or even LCM. I am familiar with gathering logs from Windows Event Log using NXLOG however we would have to install and configure such a tool using DSC before it would be of any use...

If you can support hooking into this log pipeline and direct this data elsewhere that would be great, but it is not clear in the documentation. Perhaps there is a way to leverage the 'DependsOn' property to pass configuration to Log Resource. Here is a fake example:

Log LogglyExample
{
    Message = "This message will be piped out to a configurable Loggly Resource for processing."
    DependsOn = "[ResourceType]myLoggly" 
} 

The end goal would be to send DSC logs from the LCM's and Push/Pull servers into a centralized log aggregation system like ELK Stack or Loggly which can easily consume logs in JSON format and have powerful dashboarding tools.

That just might save us all a little time... image from an article

[BerheAbrha]

@mikehumphrey There is a DSC reporting feature that you can use to monitor changes on multiple nodes. Currently DSC reports any SET operations carried out by LCM on all the nodes. see the WMF 5 release note (http://blogs.msdn.com/b/powershell/archive/2015/08/31/windows-management-framework-5-0-production-preview-is-now-available.aspx) and associated notes. To see if this feature meets your need.

[mikehumphrey]

After looking into the WMF 5.0 Production Preview with the new DSC Reporting feature I have arrived at the following observations after a few hours of monkey hand flapping on my keyboard so I may be way off here...

Configuring the LCM is very straight forward and the sample below demonstrates sending the message stream every 30 minutes to a host running DSC Serveries. Each message with include a timestamp and the ConfigurationID which in this case would be the LCM GUID.

Settings
    {
        ConfigurationID                = "$guid"
        RefreshFrequencyMins           = 30
        RefreshMode                    = "PUSH"
        ConfigurationModeFrequencyMins = 15
        AllowModuleOverwrite           = $true
    }

Then the ReportServerWeb config shown below demonstrates that there is little or no ability to control the output format or severity of the messages.

ReportServerWeb ReportManager
    {
        ServerUrL                = "http://localhost:8080/PSDSCPullServer/PSDSCPullserver.svc"
        AllowUnsecureConnection  = $true
    }           

The biggest challenge with this workflow would be attempting to automate the processing of querying the host running the ReportServerWeb, normalize and transform these messages into a standardized format and then forwarding it onto your log aggregation framework. This may require message caching or load balancing depending on the system size and needs.

My Opinion: It would be a huge step forward and makes a ton of sense to at least support shipping these to the Azure Event Hub which can be used as a message bus and forwarder for any log aggregation system.

[johlju]

@mikehumphrey today both Log resource and LCM logs locally, which would mean the best approach would be to use a tool that can be configured to know how to aggregate the logs from the DSC nodes. If the Log resource would benefit from adding "log shipping" functionality, then I suggest adding a new resource to either the resource module PSDscResources or xComputerManagement (soon to be ComputerManagementDsc), not sure which one is the better one.

@mgreenegit improved log management for the LCM, is that something that can be taken in consideration going forward?

[johlju]

I asked if this could be considered for a future LCM - it might or might not be considered. I suggest you add your proposal as a uservoice, if it doesn't already exist, so that you can track it and other users can vote on it.

See also previous comment about adding a Log resource do one of the existing resource modules. I'm closing this issue at this time since we can't do anything more about it in this repository.