Introduce a resource that enumerates files inside a zip archive

[obones]

Current Terraform Version

Terraform v1.0.11
on windows_amd64

Use-cases

My CI/CD system is bundling various parts of the AWS application into zip archives. When this is meant for a lambda function, I can simply use that zip file as the source for the lambda But I also need to place the content of some zip files into an S3 bucket to serve as the source for an API gateway integration. It would thus be nice to be able to use a construct that serves as the source for a for-each statement applied to a aws_s3_bucket_object resource

Attempted Solutions

If the zip archive has its files already extracted locally, I can achieve the for-each part just fine by using the hashicorp/dir/template module like this:

module "template_files" {
  source = "hashicorp/dir/template"

  base_dir = "${path.root}/doc"
}

resource "aws_s3_bucket_object" "doc_files" {
  for_each = module.template_files.files

  bucket       = aws_s3_bucket.doc.id
  key          = each.key
  content_type = each.value.content_type

  source  = each.value.source_path
  content = each.value.content

  etag = each.value.digests.md5
}

Note that I don't really need the "template" part, I just use the enumeration of files here.

Extracting the files can be done beforehand with a null_resource and an associated local-exec provisioner that would retrieve the zip file and then unzip it at the appropriate location

Proposal

Using a local-exec provisioner means the terraform configuration depends on operating system particularities where it runs, which makes it less portable. The documentation itself warns against such a use case:

Provisioners should only be used as a last resort

This is why I believe it would be nice to have a module that would enumerate the content of a zip file, thus allowing to have everything defined in Terraform without relying on external scripts that depend on the operating system where terraform is run.

References

https://discuss.hashicorp.com/t/enumerate-zip-file-content/32785/3

[crw]

Thanks for translating your feature request from Discuss! If there are any updates we will update this issue.

[apparentlymart]

Thanks indeed for bringing this over to be a feature request!

I do want to be up front here and note that the team which works in this repository is not a team that typically develops or maintains individual Terraform providers, but since this is entirely new functionality where it's not really clear which provider it might belong to or whether it ought to be an entirely new provider that doesn't exist yet, this does seem like a reasonable place to keep the use-case recorded for now.

One possible candidate answer would be to add this to the hashicorp/archive provider, which already contains functionality for generating archives and so it already has the relevant dependencies for interacting with archive files. However, that would still be an increase in its scope and I'm not the one to make the call about whether that scope increase would be appropriate, and so I think we'll need to let the maintainers of that provider weigh in to see if they think it makes sense, or if a different approach would be preferred.

[obones]

No worries, I'm usually on the other side of the feature request, so I always take the time to write one as I'd like to receive it.

I have no issue with the triage process, it's perfectly understandable. I mean, if in the end this is accepted, I'm happy with it, wherever it may be placed.

[obones]

Looking at this issue from the AWS provider, I see that the fileset() function has been created to enumerate the files in a given folder, so I could get rid of the dir/template module. It does not help with reading the contents from inside a zip archive, but this could help decide where to put the functionality, maybe by introducing a zipfileset() function for instance.

[apparentlymart]

Assuming you're talking about hashicorp/dir/template, that module is in fact a higher-level wrapper around that same fileset function, and so indeed you can get all of the same results yourself in terms of that function, or get a simpler result if you don't need any of the extra functionality that module adds on top.

I don't think we'd consider adding any built-in functions to Terraform which interact with specific archive file formats, because that's a pretty heavy set of new code to introduce for a relatively specialized need. However, if we implement something to address #25940 in future then that would in principle allow a provider plugin to offer a function like that, which would be essentially the same as a data source but with a subjectively "lighter" call syntax. We do have a high-level initial design proposal for that internally but haven't yet been able to prioritize a detailed exploration of it.

It is interesting to think about what effect the addition of a means to enumerate zip file contents would have on the design of a module like hashicorp/dir/template. Currently that module directly accesses the filesystem itself and so it wouldn't be possible to reuse it to get a similar effect based on the contents of a zip file, and so it seems like that would encourage a dependency-inversion-like approach where the caller passes in the set of files to render, and there might be other sibling modules which encapsulate the behavior of building a suitable set from various different sources. Of course, whether that single module being replaced by a suite of modules that need to be wired together by the caller would be a net win is debatable; the main value of that module as it exists today is it packaging together a specific set of functionality that meets a particular specialized use-case.

[apparentlymart]

FWIW, Terraform v1.8 has now introduced the possibility for providers to contribute new functions to the Terraform language, and so it's now technically possible for someone to write a utility provider that includes functions for working with zip files.