RFC for pluggable logging model

[joeyaiello]

Per #106, this is to track the need for an RFC that would create a pluggable provider model for logging. Quoting me from that thread:

@PowerShell/powershell-committee discussed this one in detail today. We believe that there's massive value in figuring out how to tee logs off to remote logging providers, but a bunch of the semantics in this RFC are specific to Splunk.

Instead, we think an RFC should be authored that builds a provider/plugin model for allowing multiple remote logging providers. This RFC should also give consideration to whether local logging targets should be treated the same way (e.g. if you want to exclusively log to a remote target, maybe in serverless/stateless scenarios). There should also be a consideration of whether we should have something like a Get-PSLog or Write-PSLog that are agnostic to logging providers.

The PowerShell Team has a strong interest in picking this up, but we have no idea when we can get to it right now. We only know that it should be a flexible, pluggable model.

[rhysjtevans]

A provider/plugin model sounds good if the resource were there to support it, however I feel it is over engineering the problem and the point was missed. Personally I'm not sure the community adoption would be there to support building plugins. Which building this in to PowerShell natively would be a wasted effort.

The original RFC (https://github.com/PowerShell/PowerShell-RFC/pull/106) was to be able to push logs to a logging platfrom (to aleviate concern Splunk was only an example).

The idea was to keep it simple,

• Both Splunk and Logstash (component ofthe ELK stack) have TCP inputs that accept RAW TCP plaintext (also over TLS), although the idea would be to structure the data e.g. k/v/json
• UDP is intended to support syslog receivers
• HTTPS is intdended to support layer 7 only load balancing environments

[Jaykul]

I think you could count on many different people wanting to be involved -- there are multiple modules each for writing data to NLog and Log4Net, as well as modules for Gelf, Sentry, and even for LogAnalytics and Splunk...

There are also a lot of implementations of custom loggers on ASP.NET Core.

The logical way to do this would be to use Microsoft.Extensions.Logging.Abstractions and the ILogger<T> and ILoggerFactory, and add support for configuration at the runspace level.

Perhaps we could then add a Set-Logging command that would let people configure an ILoggingBuilder in the machine profile, something like a scriptblock parameter implementing the Action<ILoggingBuilder> in ASP.NET's ConfigureLogging ...

In an ideal world, I would just want to be able to ship machine-level profile scripts that import (for example) a PSNLog module or a Splunk module, or the LogAnalytics module, and have the module call Set-Logging and configure logging to go to their library ... and then leave it up to the module and the library to support configuration of the formatting, and where the logs actually end up (as each of those asp.net custom logger implementations has a slightly different way of configuring the formatting, output, etc).

Once all that is in place, then certainly we would want to be able to log not just transcriptions, but also (and separately) streams of Verbose, Information, Warning, and Error, etc., and perhaps even add a new Write-Log command that sends data only to the ILogger ...

[FriedrichWeinmann]

Sounds like the kind of topic I'd be interested in :) Just for reference - I know I have bragged ab... :cough: talked about it to Joey and Joel already - I've already got a plugin-capable logging system up and available in the PSFramework module. I invite everybody interested in logging to take a peek at how it's set up, will be presenting about it at the Summit this year.

From a design perspective, one thing that is important to me is to ensure, that the Write-Log command (whatever its final name would be) is agnostic to the logging plugin(s) used. This will indeed mean that some SIEM solution's language features will only see partial utilization, but at the same time ensure, that all scripts remain interchangeable - I could use another person's script and tie it into my logging without any adjustments. Otherwise, this could be triggering lots of undesired forks.

Also @Jaykul : I'm not sure at all whether Runspace-specific configurations are needed. Usually, when I use Runspaces they are part of the main Runspace's logic workflow (e.g. Scaling out workloads) that should all be captured by the same log. Possibly as a secondary option, but process-wide should be the default in my opinion.

Multiple plugins enabled in parallel: It hasn't been said yet, but I think multiple plugins should be able to coexist in an enabled state - e.g. logging both to Splunk and Graylog at the same time. This would be useful from a migration perspective, but also when you have to feed the central system but also want to log it somewhere your own team has control over.

Asynchronous logging. Pretty sure that was implied already in the example logging solutions above, but want to explicitly mention it. Logging to network providers synchronously would obviously be a significant drag on script execution. Implied at the same time would be a way to flush the logs before the process can terminate.

[markekraus]

Re: Runspace-specific configurations

I can see at least one use-case: "Multi-tenant" hosted PowerShell environments.

[Jaykul]

I didn't mean the runspace config to be exclusive, what I was really after is a central place where it can be configured that will affect everything, and I now realize that I was mistaken to suggest configuring it in the profile was enough.

If we're talking about Transcripts, this is about to security, so:

1. The default configuration must be at the machine level, and must not be replaced or removed at lower scopes.
2. Obviously it must be configurable via powershell.config.json, because people creating runspaces for malicious reasons won't run profiles.

For the same reason, we probably should not have a Set-Logging that gives you direct access to ILoggingBuilder like ASP.NET's ConfigureLogging does, but maybe only an Add-Logger -- that is, perhaps lower scopes should be allowed to add loggers, but not remove loggers set at higher scopes?

Maybe we're not even talking about a single feature. Do we need the security feature (setting up ILoggers for Transcripts) and the debugging feature (settings up loggers for processes, runspaces, modules, or scripts) to be separate, or can they all just be ILogger with different verbosity levels? My initial preference is for a single feature, so everything works "the same way" ... but maybe we need two totally different loggers to avoid cluttering the security transcripts with "verbose" output.

Perhaps there's a transcript LoggerFactory that's only configurable in config.json, and a separate debugging 'LoggerFactory' that can be overriden and re-configured?

[rhysjtevans]

@Jaykul Always happy to be proven wrong 😄, Spot on, I typically work with a zero trust mindset - Logs should be immutable/considered a source of truth. If at runtime a script could change the logging config which defeats the purpose of a security log.

Also, some other considerations that are worth a mention

• Should be able to cope with immutable storage (i.e. not relying on underlying storage)
  • e.g. AWS Lambda only provides rw access to /tmp
• Work cross-platform

When initially opening the PR, I presumed something could be implemented on top of this https://github.com/PowerShell/PowerShell/blob/d80154430d600a52a1d1e3a9f3ecf032b36185d4/src/System.Management.Automation/logging/LogProvider.cs#L283

Although not sure if that would be adequate based on previous comments?

[TravisEz13]

I think about this more about ScriptBlock Logging than Transcripting. We could possibly extend it to include the transcripting features in the future. Right now, I haven't done enough research to have any definitive thought about the extensibility model. Here are my current thoughts:

1. If going over the network, it must be encrypted by default.
   • Using non-encrypted traffic must have a specific Opt-In
2. I agree with @rhysjtevans, the logs should be consider an immutable source of truth.

[rhysjtevans]

My thoughts were definitely leaning more towards scriptblock logging. I haven't seen anything around it for PowerShell Core though especially cross-platform availability or if there's any plans for an AMSI-like port to core.

[TravisEz13]

AMSI like logging might also be interesting. Transcription is more like logging as the console logs it.

[dragonwolf83]

I would want the existing Streams and Write cmdlets/functions to work automatically with the Logger without having to use a separate Write-Log -Verbose to log entries. That keeps it more consistent and compatible with most scripts already written.

Are there any Logging details that truly need to be handled with the message? Is it common enough that it can easily just be added to the Write cmdlets without adding log provider specifics? Dates can be automatically appended to the message with a global format and let the Logger determine where to place them. Log Level is already defined by which Stream you write to. What else is missing?

[dragonwolf83]

Security logging is certainly separate concern from debug and process logging. Transcription didn't solve logging for scripters because it's too much, hard to read, and doesn't give control for what to show.

While Log Provider needs to handle both types of logging, I think an RFC needs to first focus on the debug and process logging for users because that is a big feature missing. If security is tackled at the same time, the RFC needs to be clear that they are 2 separate audiences with different needs.

For example, security logging should always be set and locked down by admins. Process and debug logging needs to be controlled by script, profile, json config, maybe environment variables with option to allow non-admins to change it.

[SteveL-MSFT]

As mentioned, I believe there is a need for both types of logging, but we should keep the discussion separate as I agree that the use cases for both are distinct. We should use this issue to discuss script execution logging and, if needed, open a separate issue to discuss script block/AMSI style logging.

[rhysjtevans]

Totally agree re. process / verbose logging views. I would, however, be concerned around prioritization. The RFC I originally wrote was for the script block/AMSI style logging as that currently does not exist.

@dragonwolf83 For debugging / process logging purposes it's trivial to write your own module/function that could override Write-Log (as an example) or use what's already out there e.g. @FriedrichWeinmann PSFramework or PSNLog providing app-specific details.

On a tangent - I'm very tempted to start a project that would essentially wrap NLog.Targets.* (more research required). I feel that would be interesting despite the associated pro's / cons. What are your thoughts? If anyone is interested in collaborating feel free to reach out or have any suggestions on the implementation would be great to hear.