Reduce noise in logs for auditing/diffing

hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.

http://www.packer.io

Other

15.12k stars 3.33k forks source link

Reduce noise in logs for auditing/diffing #8292

Open ciarancourtney opened 5 years ago

ciarancourtney commented 5 years ago

Feature Description

When doing versioned, deterministic (as much as it is possible installing OSs) builds, its useful to store the build log output, but this output can contain lines that can change every time, creating noise that has to be ignored, incl:

SSH port line e.g. Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 2374)
Uploading randomly named scripts e.g. Provisioning with shell script: /tmp/packer-shell466354443
File upload/download progress e.g. file.tar 5.53 MiB / 13.14 MiB [==============
Different path separators if building on windows/nix

I guess it could be an opt in arg/var like like --less-log-noise

SwampDragons commented 5 years ago

Hi, thanks for reaching out! As you've seen from the above output, this isn't really how we've used logs in the past, so I am wondering if there's some other feature that would solve your problem more cleanly. What specific information would be useful to you for the diffs you are trying to collect? It sounds to me like you want some kind of audit record for what scripts were run, etc. Is that accurate?

ciarancourtney commented 5 years ago

Really its the raw output of adhoc scripts, some of which spew verbose debug output for auditing/diffing later. So I'd like to use the overall build log as a log aggregater, so devs don't have to remember to explicitly setup logging for each script.

Capturing the kickstart/preseed output would be great as well, but I don't think this is something packer or even HVs like vbox can handle, in such cases the only option is to download local logs from VM at end.

SwampDragons commented 5 years ago

Got it. I don't think this feature is something I'm going to be able to prioritize because it would likely represent a very large amount of tedious work for a reasonably small gain (I think most people end up using tools like inspec when they want to audit that builds ran successfully, rather than diffing run logs)

That said, it shouldn't be too hard to change your execute_command to pipe stdout and stderr to a file on your vm, then use the file provisioner to download stored stdout for future diffing.