Unreadable module directory when module is .tbz2 archive

mkyc commented 4 days ago

Terraform Version

Terraform v1.5.7

Terraform Configuration Files

irrelevant

Debug Output

irrelevant

Expected Behavior

have module in s3

module "vm_xxx" {
 source = "s3::https://s3-eu-west-1.amazonaws.com/xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.0.tbz2"

  ...
}

run terraform init:

...
Downloading s3::https://s3-eu-west-1.amazonaws.com/xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.0.tbz2 for vm_xxx...
- vm_xxx in .terraform/modules/vm_xxx
...

be happy

Actual Behavior

I moved some of my modules to S3 as they can be served from there as described here.

As described archive can be of different extension as described in section above.

But when archive is in .tbz2 it doesn't work. With zip it's ok.

Steps to Reproduce

have module

module "vm_xxx" {
 source = "s3::https://s3-eu-west-1.amazonaws.com/xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.0.tbz2"

  ...
}

run terraform init:

╷
│ Error: Unreadable module directory
│ 
│ The directory .terraform/modules/vm_xxx could not be read. This is a bug in Terraform and should be reported.
╵

there is file:

ls -la .terraform/modules/vm_xxx
-rw-rw-rw-@ 1 xxx  staff  5005 Nov 22 15:56 .terraform/modules/vm_xxx

and it is archive because I can extract it manually, but it doesn't get extracted by terraform.

With zip it works.

Additional Context

content of those two packages:

➜  aws s3 cp s3://xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.1.zip .
download: s3://xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.1.zip to ./aws-ec2-mssql-0.3.1.zip
➜  aws s3 cp s3://xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.0.tbz2 .
download: s3://xxx-bucket/aws-ec2-mssql/aws-ec2-mssql-0.3.0.tbz2 to ./aws-ec2-mssql-0.3.0.tbz2
➜  tar -tvjf aws-ec2-mssql-0.3.0.tbz2 
drwxrwxrwx  0 root   root        0 Nov 22 12:43 ./
-rw-rw-rw-  0 root   root     3865 Nov 22 12:43 ./main.tf
-rw-rw-rw-  0 root   root       64 Nov 22 12:43 ./outputs.tf
-rw-rw-rw-  0 root   root      288 Nov 22 12:43 ./versions.tf
-rw-rw-rw-  0 root   root       58 Nov 22 12:43 ./locals.tf
-rw-rw-rw-  0 root   root     1606 Nov 22 12:43 ./data.tf
-rw-rw-rw-  0 root   root      419 Nov 22 12:43 ./variables.tf
➜  unzip -t aws-ec2-mssql-0.3.1.zip  
Archive:  aws-ec2-mssql-0.3.1.zip
    testing: main.tf                  OK
    testing: outputs.tf               OK
    testing: versions.tf              OK
    testing: locals.tf                OK
    testing: data.tf                  OK
    testing: variables.tf             OK
No errors detected in compressed data of aws-ec2-mssql-0.3.1.zip.

References

No response

bschaatsbergen commented 4 days ago

Hey @mkyc,

Thanks for reporting this! Based on our documentation on module sources and supported archives (https://developer.hashicorp.com/terraform/language/modules/sources), I'd expect your use case to be valid and working.

I'm able to reproduce your issue using:

module "example" {
 source = "s3::https://36080.s3.us-west-2.amazonaws.com/example.tbz2"
}

From a quick look, it seems that Terraform is successfully downloading the archive into the .terraform directory:

2024-11-22T18:27:33.851+0100 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
2024-11-22T18:27:33.851+0100 [DEBUG] Module installer: begin example
2024-11-22T18:27:33.851+0100 [TRACE] ModuleInstaller: example is not yet installed
2024-11-22T18:27:33.851+0100 [TRACE] ModuleInstaller: cleaning directory .terraform/modules/example prior to install of example
2024-11-22T18:27:33.851+0100 [TRACE] ModuleInstaller: example address "s3::https://36080.s3.us-west-2.amazonaws.com/example.tbz2" will be handled by go-getter
Downloading s3::https://36080.s3.us-west-2.amazonaws.com/example.tbz2 for example...
2024-11-22T18:27:33.851+0100 [TRACE] getmodules: fetching "s3::https://36080.s3.us-west-2.amazonaws.com/example.tbz2" to ".terraform/modules/example"
2024-11-22T18:27:34.809+0100 [TRACE] ModuleInstaller: example "s3::https://36080.s3.us-west-2.amazonaws.com/example.tbz2" was downloaded to .terraform/modules/example
2024-11-22T18:27:34.809+0100 [DEBUG] Module installer: example installed at .terraform/modules/example

It seems the download is successful, but something further down the line is causing an issue. I noticed the downloaded archive is missing a file extension, which is causing the “Unreadable module directory” error diagnostic to surface.

 $ tree .terraform
.terraform
└── modules
    ├── modules.json
    └── example

FWIW, uploading a copy of my archive, using a tar.tbz2 extension seems to work.

- source = "s3::https://36080.s3.us-west-2.amazonaws.com/example.tbz2"
+ source = "s3::https://36080.s3.us-west-2.amazonaws.com/example.tar.tbz2"

It’s possible that the documentation referencing the tbz2 file extension, is currently not supported.

From my understanding—though feel free to correct me if I’m wrong—tbz2, tar.tbz2 and tar.bz2 use the same compression algorithm (bzip2), with tbz2 being just a shorthand for the longer tar.tbz2 and tar.bz2 extensions[¹].

For clarity and to avoid confusion:

tar.bz2, tar.tbz2, tbz2: Tar archives compressed with bzip2.
bz2: General bzip2-compressed file.

In go-getter, the library Terraform uses to download files and directories from various sources, there is a decompressor for tar archives compressed with bzip2 that can handle both tbz2, tar.tbz2 and tar.bz2 files[²].

It seems we may have overlooked support for the tbz2 archive extension, causing the fetching process (via the getter) to succeed but leading to a decompression failure. This results in an unexpected file without an extension.

I’ll provide an update here as soon as I have more information. Thanks again!

mkyc commented 4 days ago

Hi @bschaatsbergen,

thanks for solid investigation! I think you ticked all the boxes with your explanation 👍

hashicorp / terraform