Issue: Output is null

[annedroiid]

I'm using v1.0.0 of the provider with terraform 0.12.24 and I keep getting an error. This happens both on a linux executor on CircleCI and locally on my Mac using macOS 10.14.

I've created a script to generate a kafka keystore/truststore but I keep getting this error when I try to reference the output of the script

Error: Attempt to index null value

  on infrastructure/gcp/modules/common/kafka-config/config.tf line 10, in resource "kubernetes_secret" "credentials":
  10:     keystore                 = replace(shell_script.create_kafka_stores.output["base64_encoded_keystore"], "\n", "")
    |----------------
    | shell_script.create_kafka_stores.output is null

This value is null, so it does not have any indices.

Here is my resource

resource "shell_script" "create_kafka_stores" {
  lifecycle_commands {
    create = file("${path.module}/create_stores.sh")
    delete = "echo {}"
  }

  environment = {
    ID = random_integer.id.result
    KAFKA_PRIVATE_KEY        = ovo_kafka_user.service_user.access_key
    KAFKA_CLIENT_CERTIFICATE = ovo_kafka_user.service_user.access_cert
    KEYSTORE_PASSWORD        = random_string.ssl_password.result
    TRUSTSTORE_PASSWORD       = random_string.ssl_password.result
    KAFKA_SERVICE_CERTIFICATE = var.kafka_ca_certificate
  }
}

and here is the script I'm using

#!/bin/sh

# Exit if any of the intermediate steps fail
set -e

echo "$KAFKA_SERVICE_CERTIFICATE" > /tmp/ca.txt

echo "$KAFKA_PRIVATE_KEY" > /tmp/compositefile-$ID.txt
echo "$KAFKA_CLIENT_CERTIFICATE" >> /tmp/compositefile-$ID.txt
openssl pkcs12 -export -in /tmp/compositefile-$ID.txt -out /tmp/keyStore-$ID.p12 -password pass:$KEYSTORE_PASSWORD

#`python -m base64 -d` base64 decoding to be able to run on Mac and Linux the same way
echo "$KAFKA_SERVICE_CERTIFICATE" | python -m base64 -d | keytool -keystore /tmp/trustStore-$ID.jks -alias CARoot -import -storepass $TRUSTSTORE_PASSWORD -noprompt

KEYSTORE_BINARY=$(base64 /tmp/keyStore-$ID.p12)
TRUSTSTORE_BINARY=$(base64 /tmp/trustStore-$ID.jks)

jq -n --arg keyStoreBinary "$KEYSTORE_BINARY" --arg trustStoreBinary "$TRUSTSTORE_BINARY" '{"base64_encoded_keystore":$keyStoreBinary, "base64_encoded_truststore":$trustStoreBinary}' >&1

Is there something different I should be doing to get the outputs from the resource? The kubernetes_secret in which I'm referencing the output also has a depends_on block to depend on the shell resource

depends_on = [shell_script.create_kafka_stores]

[scottwinkler]

Hello Anne, In version 1.1 of the provider there were significant changes made to the way complex json is handled by the output (meaning you no longer have to have a map[string]string, you can output any arbitrary json) and in addition, I fixed a bug that caused json output not to be read properly. Could you please download the latest version of the provider and try again? Thanks.

[annedroiid]

I'll try updating to 1.1 now 😊

Looking at more of the output, I'm not sure that the script is actually being run before it tries to reference the output. I added a sleep 30 into the script and terraform still tells me

shell_script.create_kafka_stores: Creating...
shell_script.create_kafka_stores: Creation complete after 0s 

Do you know if this would be an issue with terraform or with the provider?

[annedroiid]

Just ran it again with 1.1 and it still has the same issue.

If I target the resource itself terraform successfully runs, but it doesn't look like it has stored the output. When I then try to run terraform again it says the resource needs to be recreated as there is no output

  # shell_script.create_kafka_stores must be replaced
-/+ resource "shell_script" "create_kafka_stores" {
        dirty             = false
        environment       = {
            "ID"                        = "38449"
            "KAFKA_CLIENT_CERTIFICATE"  = <<~EOT
                -----BEGIN CERTIFICATE-----
                my-certificate
                -----END CERTIFICATE-----
            EOT
            "KAFKA_PRIVATE_KEY"         = <<~EOT
                -----BEGIN PRIVATE KEY-----
                my-private-key
                -----END PRIVATE KEY-----
            EOT
            "KAFKA_SERVICE_CERTIFICATE" = "my-ca-certificate"
            "KEYSTORE_PASSWORD"         = "password"
            "TRUSTSTORE_PASSWORD"       = "password"
        }
      ~ id                = "bqg7eubmvbaobe40o750" -> (known after apply)
      + output            = (known after apply) # forces replacement
        working_directory = "."

      ~ lifecycle_commands {
            create = <<~EOT
                #!/bin/sh

                # Exit if any of the intermediate steps fail
                set -e

                echo "$KAFKA_SERVICE_CERTIFICATE" > /tmp/ca.txt

                echo "$KAFKA_PRIVATE_KEY" > /tmp/compositefile-$ID.txt
                echo "$KAFKA_CLIENT_CERTIFICATE" >> /tmp/compositefile-$ID.txt
                openssl pkcs12 -export -in /tmp/compositefile-$ID.txt -out /tmp/keyStore-$ID.p12 -password pass:$KEYSTORE_PASSWORD

                #`python -m base64 -d` base64 decoding to be able to run on Mac and Linux the same way
                echo "$KAFKA_SERVICE_CERTIFICATE" | python -m base64 -d | keytool -keystore /tmp/trustStore-$ID.jks -alias CARoot -import -storepass $TRUSTSTORE_PASSWORD -noprompt

                KEYSTORE_BINARY=$(base64 /tmp/keyStore-$ID.p12)
                TRUSTSTORE_BINARY=$(base64 /tmp/trustStore-$ID.jks)

                sleep 10

                jq -n --arg keyStoreBinary "$KEYSTORE_BINARY" --arg trustStoreBinary "$TRUSTSTORE_BINARY" '{"base64_encoded_keystore":$keyStoreBinary, "base64_encoded_truststore":$trustStoreBinary}' >&1
            EOT
            delete = "echo {}"
        }
    }

key line being

+ output            = (known after apply) # forces replacement

[annedroiid]

I'll try updating to 1.3 and see if it still happens.

[scottwinkler]

Can you please provide a copy of the json you are outputting? I dont need any of the secrets, just the structure of the json. I notice you also have "set -e". Is it possible one of the intermediate steps has errors and that is why you arent seeing the script run? Can you set TF_LOG=debug to print execution logs?

[annedroiid]

I hadn't thought about that, I'll remove that line and add debug and see what it outputs.

[annedroiid]

The format of the json is the output of the jq line, so

{"base64_encoded_keystore":$keyStoreBinary, "base64_encoded_truststore":$trustStoreBinary}

[scottwinkler]

It may not be properly formatted json if the variables dont have quotes around them. Have you tested this script manually to verify that it outputs proper json?

[annedroiid]

Yes when you reference a variable like that it's smart enough to put the quotes around it for you.

After getting rid of set -e it's now working, there must have been some sort of inconsequential error that was happening during execution. Thank you very much for your help, this looks like it's an error with my script and not with the provider.

[annedroiid]

@scottwinkler On a related note, is the resource meant to get created even if the script fails? I would have expected that the resource creation would fail if the script doesn't run successfully.

[scottwinkler]

Messages to stderr do not cause a resource to fail because sometimes commands can output to stderr without it really being a problem. If you would like a resource to fail then return a non zero exit code from the script and that will throw a hard error in Terraform

[annedroiid]

Gotcha, thanks 😊