The apply-all and destroy-all commands should save log output to file

[brikis98]

When you run terragrunt apply-all or terragrunt destroy-all, they can make changes across a ton of different modules, in parallel. This is great, but as mentioned in #71, the amount of log output is overwhelming, and it's easy to miss something important. One small step in helping with this is to store the log output for each module somewhere. Perhaps within the module or a tmp directory will do.

[justyns]

Posting this here in case it helps someone else. You can save the output of the terraform commands by making a wrapper around terraform and having terragrunt call that.

An example wrapper script:

#!/bin/bash

terraform $@ 2>&1 | tee ./terraform.log

And then using it with terragrunt:

terragrunt apply-all --terragrunt-non-interactive --terragrunt-tfpath terraform-wrapper

For me, this creates a terraform.log file in each directory where terragrunt plan/applied something.

Ideally terragrunt could do something like this by itself, but this workaround helps until that happens.

[falmotlag]

Solution

Delay Terraform output for all modules and display it formatted at the end of Terragrunt execution.

CLI option

Add a --delay-terraform-output cli option to turn on this feature.

Code Change

1. Add a --delay-terraform-output cli option to turn on this feature.
2. in stack.Plan and stack.Apply assign a buffer to Writer
3. defer to function that print buffer of each module

Snippet of change to stack.Plan(terragruntOptions *options.TerragruntOptions) error:

if terragruntOptions.DelayOutput == true {
    outputStreams := make([]bytes.Buffer, len(stack.Modules))
    for n, module := range stack.Modules {
        module.TerragruntOptions.Writer = &outputStreams[n]
    }
    defer stack.printDelayedOutput(terragruntOptions, outputStreams)
}

Code to create cli flag and to printDelayedOutput is excluded from the snippet.

I am not too familiar with the codebase so I could be missing something here. Can someone familiar with the codebase review this?

[brikis98]

Thx for the proposal!

Unfortunately, I've often found that buffering log output for a long period of time is not a great solution for a few reasons:

1. With no output, the user has no idea what's going on. Is apply still running? Is there an error? What's being deployed? This leads to hitting CTRL+C, which is not a good combination with Terraform.
2. Some CI systems will kill a build if it goes more than N minutes without log output (e.g., in CircleCi, the limit is 10m).

Therefore, I think we'd have to stream the log output to stdout/stderr, and if there's any sort of buffer, it's a file on disk.

[yorinasub17]

I wonder if we could just buffer the stdout and print out one module at a time in a stream, but continue to stream stderr logs? Terraform is fairly consistent in outputting the useful information to stdout in a single timestep so I don't think that would be too long.

[dudicoco]

@yorinasub17 would it be possible to implement this? I think most of the terragrunt community have been waiting for years for such a feature :)

[geota]

Logging in terrgrunt is one of the reasons it has been harder than I had anticipated getting wider adoption of TG within my current company.

[acesaro]

How about a solution that simply prefixes and optionally colorizes the log lines with the component (individual terraform command) they are output from. The prefixes would allow you to tee/grep into various log files to help with the original ask of this GH issue. Packer does this really well when you run a build that includes multiple items in the sources array of an HCL2 Packer template. This is a pleasant experience with multiple builds executing in parallel.

An example of that output is shown at the end of this linked section: https://learn.hashicorp.com/tutorials/packer/docker-get-started-parallel-builds#build-images

[komal-SkyNET]

The fundamental requirement of any wrapper is to have the ability to expose logs of executed operations in a useful and understandable fashion – and Terragrunt has failed to meet this most basic requirement. This issue has been repeatedly brought up since 2016, yet there has been not a single improvement in this area.

[atheiman]

We use terragrunt and it's great, but some people might benefit from using a more generic tool like GNU parallel: https://www.gnu.org/software/parallel/parallel_tutorial.html

It lets you run a batch of commands with different arguments in parallel, very similar to terragrunt run-all approach. It also has many built in configuration options for how to log output from each parallel operation to individual files, buffer them and output one at a time to stdout, etc.

[szaher]

We hit the same issue recently, to keep the same behavior of the terraform wrapper, I modified it to print the current module name only if the action is not output.

If terraform output > don't print the module name and log output, else print the module name

#!/bin/bash
# (C) Copyright 2019-2022 Hewlett Packard Enterprise Development LP

set -o nounset
set -o errexit
set -o pipefail

TERRAFORM_ACTION=$1

# ignore printing directory name when the command is terraform output
if [[ $TERRAFORM_ACTION == "output" ]]; then
  terraform "$@" 2>&1 | tee "./terraform-output.log"
  else
    # shellcheck disable=SC2046
    terraform "$@" 2>&1 | ts "[$(basename $(pwd))]"
fi

hope it helps

[nijave]

We use terragrunt and it's great, but some people might benefit from using a more generic tool like GNU parallel: https://www.gnu.org/software/parallel/parallel_tutorial.html

It lets you run a batch of commands with different arguments in parallel, very similar to terragrunt run-all approach. It also has many built in configuration options for how to log output from each parallel operation to individual files, buffer them and output one at a time to stdout, etc.

Parallel doesn't understand Terragrunt module dependencies

[atheiman]

@nijave sure, but not everyone uses that feature. Its a workaround I suggested, not meant to be a total solution to this issue. I know a few customers I have worked with use terragrunt solely for the run-all capability. Not for module interdependencies.

[nijave]

Maybe a more robust approach is per-module buffering with the option to overwrite console output to display realtime progress (similar to how docker build or curl progress work).

i.e. --output=[unbuffered,buffered,live] unbuffered - current functionality buffered - per module buffering live - dynamic output where the last few lines per running module are shown like docker buildkit

[Aldairng]

Another idea here, what if we simply modify the output of each line of terraform log adding a prefix, that way we can identify where each line is coming from?

I don't want to lose terragrunt's parallel feature, so i discard a buffering option. I think this is an easy solution that only requiere existing utilities like sed or perl.

In the image i'm using sed for a regular run-all plan, this should be added when executing each plan in terraform. I also measured the time and it is similar.

Right now the prefix is the same all over the place (this just and example), we should use the name of the module dinamically, for example for me: efs, rds, etc.

note: tgis my alias for terragrunt