cattle-ops / terraform-aws-gitlab-runner

Terraform module for AWS GitLab runners on ec2 (spot) instances
https://registry.terraform.io/modules/cattle-ops/gitlab-runner/aws
MIT License
586 stars 331 forks source link

chore: set specific file mode for the zipped lambda function to ensure consistent checksums #1193

Closed tibuntu closed 1 month ago

tibuntu commented 1 month ago

Description

Hello, this is a follow up of the problems reported in #1171

After digging deeper into the problem, we figured out some more details.

We use Atlantis, running in a Kubernetes cluster, to plan & apply Terraform changes.

The Atlantis PVC is backed by a NFS based backend, therefore not really POSIX compliant. Our guess is that this causes the source_code_hash to flap between two different values STtJd0//CpU4WGQcc2ojckEcEHtCrD8W2fpx9E1TL64 (Which is the result if the file permissions are 0622) and 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= (Where I couldn't figure out which exact permissions cause this value to be computed. I tried almost all variantes such as 0644, 0777, etc.)

When running a plan via Atlantis the result sometimes changes within minutes. While doing the research, we found: https://github.com/hashicorp/terraform-provider-archive/issues/34 which describes the problematic behaviour. As you can see, to avoid such issues, a new argument output_file_mode was added to the provider, to omit the problem.

After adding output_file_mode to all modules during some local tests, the results were always the same. See also: https://registry.terraform.io/providers/hashicorp/archive/latest/docs/data-sources/file#output_file_mode

I am still really curious, why no one else reported such a behaviour before. It might be the combination of Atlantis / NFS storage as noted above. However, I don't believe that this change will cause any problems with the module.

Migrations required

No

Verification

Copy the lambda_function.py (Any other test file should work, too) into the same directory as this Terraform code:

locals {
  source_sha256 = filesha256("lambda_function.py")
}

data "archive_file" "terminate_runner_instances_lambda" {
  type        = "zip"
  source_file = "lambda_function.py"
  output_path = "builds/lambda_function_${local.source_sha256}.zip"
}

output "lambda_zip_shasum" {
  value = data.archive_file.terminate_runner_instances_lambda.output_base64sha256
}

Run apply to see the generated shasum as Terraform output:

lambda_zip_shasum = "STtJd0//CpU4WGQcc2ojckEcEHtCrD8W2fpx9E1TL64="

Modify the the file permissions:

chmod g+w lambda_function.py

Run Terraform again:

> terraform apply
data.archive_file.terminate_runner_instances_lambda: Reading...
data.archive_file.terminate_runner_instances_lambda: Read complete after 0s [id=e011e457b35f7615551664de5224d2f7f292a697]

Changes to Outputs:
  ~ lambda_zip_shasum = "STtJd0//CpU4WGQcc2ojckEcEHtCrD8W2fpx9E1TL64=" -> "pZAfgYlIKW/ruHEDNtnLDs9h7elRicVa2HhSA6fng9s="

[...]

Now repeat the same tests, but add output_file_mode = "0666" to the data source:

lambda_zip_shasum = "EJcKN1Aif+z+qa8NwFb+dv+O0vBRteYK2/RtTBiULxk="

Change the permissions again:

chmod o+w lambda_function.py

Terraform output doesn't change:

lambda_zip_shasum = "EJcKN1Aif+z+qa8NwFb+dv+O0vBRteYK2/RtTBiULxk="
github-actions[bot] commented 1 month ago

Hey @tibuntu! 👋

Thank you for your contribution to the project. Please refer to the contribution rules for a quick overview of the process.

Make sure that this PR clearly explains:

With submitting this PR you confirm that you hold the rights of the code added and agree that it will published under this LICENSE.

The following ChatOps commands are supported:

Simply add a comment with the command in the first line. If you need to pass more information, separate it with a blank line from the command.

This message was generated automatically. You are welcome to improve it.

kayman-mk commented 1 month ago

Gosh, was looking for this for a long time now. Have other Lambda functions which behave the same way.

Thanks for fixing it.

kayman-mk commented 1 month ago

Renamed to chore as it doesn't fix a problem for the consumer of this module.