Terraspace container performance

samjsmart commented 3 years ago

I'm not sure if this is a local issue or not but I wanted to raise it for further testing. We have noticed a fairly large performance impact when running Terraspace in docker, this is true both in our own images and the terraspace-dockerfiles images, this impact is especially noticeable on EC2. Here is a comparison of different execution times across local machines, EC2, docker, and host.	EC2 Docker	EC2	Local Docker
real 5m33.403s	real 0m44.799s	real 1m40.842s	real 0m39.626s
user 0m11.150s	user 0m26.531s	user 0m24.736s	user 0m10.913s
sys 0m1.437s	sys 0m3.276s	sys 0m13.846s	sys 0m4.580

This doesn't appear to be a resource issue either, as there is minimal usage during execution:

CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O     PIDS
9b05765250b8   quirky_spence   0.54%     1.21GiB / 7.583GiB   15.95%    300MB / 3.43MB   0B / 1.08MB   3

From execution observation, it appears that the majority of the time is spent on the build step. Happy to provide further information as required.

tongueroo commented 2 years ago

A guess in the dark. Wondering if it's filesystem performance within Docker.

The terraspace build stage reads and writes files.

When you get a chance, here's a special custom logger that will prepend timestamp to stdout.

config/app.rb

class CustomLogger < Terraspace::Logger
  def format_message(severity, datetime, progname, msg)
    out = (@formatter || @default_formatter).call(severity, datetime, progname, msg)
    "#{out}\n"
  end
end

Terraspace.configure do |config|config.logger.level = :debug
  config.logger = CustomLogger.new($stdout)
  config.logger.level = :debug
  config.test_framework = "rspec"
end

You should see something like this:

$ terraspace build
[2021-12-30T19:45:52 #5181 ]: Building one stack to build all stacks
[2021-12-30T19:45:52 #5181 ]: Building .terraspace-cache/dev/stacks/demo
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/backend.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/provider.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/modules/example/main.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/modules/example/outputs.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/modules/example/variables.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/backend.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/provider.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/main.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/outputs.tf
[2021-12-30T19:45:52 #5181 ]: Created .terraspace-cache/dev/stacks/demo/variables.tf
[2021-12-30T19:45:52 #5181 ]: Built in .terraspace-cache/dev/stacks/demo
$

That may provide more diagnostic info.

tongueroo commented 1 year ago

Closing due to lack of updates

boltops-tools / terraspace-dockerfiles

Terraspace container performance #4