imperva / dsfkit

Imperva eDSF Kit is designed to automate the deployment of DSF
MIT License
7 stars 9 forks source link

Do not "tail" the `replication.log` when running the `arbiter-setup run-replication` step #426

Closed dweberJsonar closed 2 months ago

dweberJsonar commented 4 months ago

Impetus

@jagdeep-sonar has reached out to me with a failed TF run: 9653669156. Something I noticed in the output is a lot of:

...
module.agentless_gw_hadr[0].null_resource.exec_replication_cycle_on_dr (remote-exec): 1.90      2024-06-24 23:29:38,...
module.agentless_gw_hadr[0].null_resource.exec_replication_cycle_on_dr (remote-exec): 2.15      2024-06-24 23:29:39,...
module.agentless_gw_hadr[0].null_resource.exec_replication_cycle_on_dr (remote-exec): 2.40      2024-06-24 23:29:39,...
module.agentless_gw_hadr[0].null_resource.exec_replication_cycle_on_dr (remote-exec): 2.66      2024-06-24 23:29:39,...
module.agentless_gw_hadr[0].null_resource.exec_replication_cycle_on_dr (remote-exec): 2.91      2024-06-24 23:29:39,...
...

There are hundreds of lines like this and we of course want to avoid bloating the logs :) This is happening because the run-replication command is by default, meant to be run in an interactive environment (i.e. manually triggered via a user). This command is kind of a "cheap" implementation of tail or watch and tails the last line of replication.log then prints that line to stdout with a carriage return. The problem is that TF is capturing each one of the "frames" being spit out in between the carriage returns. To handle this exact solution, I added the --no-tail flag to the command, thus allowing us to run this command in a non-interactive environment. You can see https://github.com/jsonar/disaster-recovery/blob/master/docs/arbiter/arbiter.rst#running-a-replication-cycle for more information regarding this.

dweberJsonar commented 4 months ago

Solution

The solution is to locate: https://github.com/imperva/dsfkit/blob/57453ea2d4236c188963b94f4267a7c9f647d1a2/modules/null/hadr/main.tf#L81-L86 And add --no-tail to the command on line 85. It looks like I can't open PRs against the repo without forking - otherwise I'd just edit this myself :)