search

localstack / terraform-local

Terraform CLI wrapper to deploy your Terraform applications directly to LocalStack
Apache License 2.0
203 stars 24 forks source link

Issue with `parse_tf_files`: function returning lists instead of string values in parsed Terraform configurations #39

Closed luktomaszewski closed 11 months ago

luktomaszewski commented 11 months ago

I've encountered an issue with theparse_tf_files function in a Python script designed to parse local Terraform (*.tf) files. The function uses the hcl2 library to load and parse the Terraform configuration files. However, the parsed output for certain keys (such as bucket and key in the s3 backend configuration) is returned as lists instead of single string values. This behavior causes problems in subsequent parts of the script, where these values are expected to be strings.

Version:

after configs.update(backend_config) tflocal#L211

{'bucket': ['tf-state'], 'key': ['terraform.tfstate'], 'dynamodb_table': 'tf-test-state', 'region': 'us-east-1', 's3_endpoint': 'http://s3.localhost.localstack.cloud:4566', 'iam_endpoint': 'http://localhost:4566', 'sts_endpoint': 'http://localhost:4566', 'dynamodb_endpoint': 'http://localhost:4566'}

get_or_create_bucket(configs["bucket"]) expects string as input tflocal#L308

providers.tf:

terraform {
  backend "s3" {
    bucket = "tf-state"
    key    = "terraform.tfstate"
  }

  required_version = "= 1.6.4"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
}
Traceback (most recent call last):
  File "/opt/homebrew/bin/tflocal", line 316, in get_or_create_bucket
    return s3_client.head_bucket(Bucket=bucket_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 928, in _make_api_call
    api_params = self._emit_api_params(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 1046, in _emit_api_params
    self.meta.events.emit(
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
               ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/handlers.py", line 278, in validate_bucket_name
    if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/bin/tflocal", line 417, in <module>
    main()
  File "/opt/homebrew/bin/tflocal", line 404, in main
    config_file = create_provider_config_file(providers)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/bin/tflocal", line 150, in create_provider_config_file
    tf_config += generate_s3_backend_config()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/bin/tflocal", line 217, in generate_s3_backend_config
    get_or_create_bucket(configs["bucket"])
  File "/opt/homebrew/bin/tflocal", line 322, in get_or_create_bucket
    return s3_client.create_bucket(Bucket=bucket_name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 928, in _make_api_call
    api_params = self._emit_api_params(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/client.py", line 1046, in _emit_api_params
    self.meta.events.emit(
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
               ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/handlers.py", line 278, in validate_bucket_name
    if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'list'
lakkeger commented 11 months ago

Hi @lomasz, Thanks for reaching out. Could you please provide some more information here, please? Namely:

Thank you for your help.

lakkeger commented 11 months ago

Additionally, what do you mean by?

in a Python script designed to parse local Terraform (*.tf) files.

luktomaszewski commented 11 months ago

Additionally, what do you mean by?

in a Python script designed to parse local Terraform (*.tf) files.

parse_tf_files()

luktomaszewski commented 11 months ago

Hi @lomasz, Thanks for reaching out. Could you please provide some more information here, please? Namely:

  • What version of terraform-local you are running?

  • Could you please provide the steps to reproduce the above error? As I was unable to on a system with the following versions:

    • python: 3.11.5
    • terraform: 1.6.4
    • terraform-local: 0.16.0
    • python-hcl2: 4.3.2

What version of terraform-local you are running?

0.16.0.

I've tried to reproduce on my second machine (both Macbook M1) with the same setup and there this error doesn't occur. 😕

lakkeger commented 11 months ago

Ok, no problem. Please let me know if you could reproduce the issue and we'll look into it. For the meantime I close this ticket.

luktomaszewski commented 11 months ago

I have successfully pinpointed the problem. It stems from a conflict between Python packages. Specifically, the issue was caused by having two distinct hcl2 parsers installed via pip, both sharing the same name. On my system, the bc-python-hcl2 package is installed alongside the checkov tool.

pip list | grep hcl
--------------------------
bc-python-hcl2            0.4.1
python-hcl2               4.3.2

pip show bc-python-hcl2
--------------------------
Name: bc-python-hcl2
Version: 0.4.1
Summary: A parser for HCL2
Home-page: https://github.com/bridgecrewio/python-hcl2
Author: bridgecrew
Author-email: meet@bridgecrew.io
License: MIT
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: lark
Required-by: checkov