Add zip support to concourse

[lukaspour]

Hi, we need a support to zip files to add requirements to our lambda code and zip it all together. What would be the best approach? Install zip package via terraform.sh or create custom docker image with zip package?

@itsdalmo whats your opinion on this please?

[mikael-lindstrom]

We have a container that includes zip which we are using for similar things. Its available on dockerhub. Its a pretty lightweight container and should be fast to use.

[itsdalmo]

Also, the image is likely to be cached on the workers since it's used for many other tasks 😄

[lukaspour]

The thing is that we directly reference zip from terraform:

resource "null_resource" "zip_append" {
  provisioner "local-exec" {
    command = "cd ${path.module}/lambda_services_dashboard && zip -ur ${path.module}/lambda_services_dashboard.zip requirements.txt main.py templates/dashboard.json.j2; rt=$?; [ $rt = 12 ] && exit 0 || exit $rt;"
  }
  triggers = {
    source_file = "${base64sha256(join("", list(file("${path.module}/lambda_services_dashboard/main.py"), file("${path.module}/lambda_services_dashboard/templates/dashboard.json.j2"))))}"
  }
}

The reason why is because we need to append some files into zip archive. Terraform provider archive_file does not allow appending the files. Also, there is check for return code 12 which means there is nothing to be done (files in zip archive are the same as the one appended to it). Terraform would consider it as bad return code, that's why there is workaround: [ $rt = 12 ] && exit 0 || exit $rt;

Can you please give me some tip how to solve this? Couple ideas:

• create a new docker with terraform and zip
• add zip instalation to terraform.sh script
• add zip installation in pipeline.yml in our neo-aws repository

[lukaspour]

@itsdalmo what approach sounds the best for you? I believe the custom docker based on terraform is the cleanest solution.

[itsdalmo]

@lukaspour - I think the cleanest solution is to avoid shelling out from terraform (by using local-exec) since it makes assumptions about the environment (and shell) it is running in, whether it is in a CI/CD or locally on someones computer. If we added zip, presumably we should add bash, and why not just add unzip/jq/awscli while we are at it - where do we draw the line?

Imo, it is cleaner to just avoid shelling out and expect the path to an existing zip file to be passed into the terraform module. Then you can add a separate task to zip the python code before running terraform apply using this concourse-terraform-task?

[lukaspour]

I wouldn't even use any external app if there would be an option to append the files to a zip archive in archive_file provider. The use-case is to have python scripts with the code we need to change and to keep modules in separate zip file. During the development we can edit those python files and append them to the archive with modules with terraform apply. That was one of the way how to keep the lambda function editable and how to keep modules as a binary blob with very little overhead on repository management. @itsdalmo do you have an other idea how to solve the issue?

I would like to avoid adding the modules to the repository if possible, because at the worse scenario, it would end up as this: https://github.com/itsdalmo/tf_aws_ecs_instance_draining_on_scale_in/tree/master/index

Having the concourse task would be great, but wouldn't help that much during development on local machine.

[itsdalmo]

Sorry I did not answer you on this one @lukaspour - I'll close the issue since its so old, and under the assumption that you resolved this already, if not you can always re-open it 👍