Fix stack cleanup

[adilhafeez]

On some instances TemplateDescription can have '\n' encoded inside it but when we process it through bash echo then it would cause line break resulting in invalid json.

Here is sample output I received from list-stacks that resulted in breaking cleanup.sh (notice \n in json response)

➜  aws-saas-factory-ref-solution-serverless-saas git:(adil/fix_stack_cleanup) ✗ saas aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE IMPORT_COMPLETE IMPORT_ROLLBACK_COMPLETE
...

        {
            "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
            "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
            "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
            "CreationTime": "2023-04-27T21:02:43.084000+00:00",
            "StackStatus": "CREATE_COMPLETE",
            "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
            "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }
        },
...

[adilhafeez]

@suhussai

[suhussai]

Hi @adilhafeez ,

I'm having trouble replicating the issue.

I copied the output in your comment and ran jq to parse it, and it didn't throw any error:

❯ cat jsonNewLine
{
  "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
  "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
  "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
  "CreationTime": "2023-04-27T21:02:43.084000+00:00",
  "StackStatus": "CREATE_COMPLETE",
  "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "DriftInformation": {
    "StackDriftStatus": "NOT_CHECKED"
  }
}

❯ cat jsonNewLine | jq
{
  "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
  "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
  "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
  "CreationTime": "2023-04-27T21:02:43.084000+00:00",
  "StackStatus": "CREATE_COMPLETE",
  "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "DriftInformation": {
    "StackDriftStatus": "NOT_CHECKED"
  }
}

❯ cat jsonNewLine | jq '.TemplateDescription'
"Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n"

I might be missing something. Can you elaborate on the exact error you are seeing and how I can reproduce it?

[adilhafeez]

here is how to repro

➜  ~ cat > json
[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
        "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
        "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
        "CreationTime": "2023-04-27T21:02:43.084000+00:00",
        "StackStatus": "CREATE_COMPLETE",
        "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "DriftInformation": {
            "StackDriftStatus": "NOT_CHECKED"
        }
    }
]
➜  ~ X=`cat json`
➜  ~ echo $X | jq .
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 6, column 1

[adilhafeez]

and the fix

➜  ~ Y=$(cat json | sed -e 's/\\n//g')
➜  ~ echo $Y
[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
        "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
        "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap",
        "CreationTime": "2023-04-27T21:02:43.084000+00:00",
        "StackStatus": "CREATE_COMPLETE",
        "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "DriftInformation": {
            "StackDriftStatus": "NOT_CHECKED"
        }
    }
]
➜  ~ echo $Y | jq .
[
  {
    "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
    "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
    "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap",
    "CreationTime": "2023-04-27T21:02:43.084000+00:00",
    "StackStatus": "CREATE_COMPLETE",
    "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
    "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
    "DriftInformation": {
      "StackDriftStatus": "NOT_CHECKED"
    }
  }
]

[adilhafeez]

This is really interesting. I have been using this cleanup.sh script for a while, and have never seen this error before. I'll have to do some digging to figure out why.

FWIW I am using mac m1 max (13.1 (22C65))

[suhussai]

This was a real head scratcher, but I think I got it. It seems to be a shell issue. I created this small script to mimic what the script was doing:

cat <<EOT >> jsonNewLine.txt
[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
        "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
        "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
        "CreationTime": "2023-04-27T21:02:43.084000+00:00",
        "StackStatus": "CREATE_COMPLETE",
        "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "DriftInformation": {
            "StackDriftStatus": "NOT_CHECKED"
        }
    }
]
EOT

response=$(cat jsonNewLine.txt)

name=$(echo "$response" | jq -r '.[0].StackName')

echo "$name"

rm jsonNewLine.txt

Now, when I run that script using bash, it works:

❯ bash script.sh
serverless-saas-APIs-17O4B7RH955WQ

But, when I run it using sh, I get the error you posted:

❯ sh script.sh
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 6, column 1

So, I think the reason I never saw this was because I typically run scripts using bash directly.

Regardless, I don't think we need to do anything else here.

Thanks again for the contribution!

[adilhafeez]

That resolves the mystery - I didn't know bash and sh treated escape characters differently. Thanks for reporting back.

[kiwi44]

Thanks for keeping up the good work!