search

aws-samples / service-catalog-engine-for-terraform-os

Apache License 2.0
130 stars 40 forks source link

Fix deploy-tre.sh #35

Closed kahilof closed 1 year ago

kahilof commented 1 year ago

Hi,

deploy-tre.sh is not working if I followed the steps here need to change path to ../../lambda-functions/terraform_open_source_parameter_parser

Thanks, Omar

zillidan commented 1 year ago

Hello Omar,

Apologies for the issue. Can you please attach the errors faced by you when deploying the TRE?. Is this related to the issue mentioned here? https://github.com/aws-samples/service-catalog-engine-for-terraform-os/issues/27

smaly-amazon commented 1 year ago

Hi,

Based on your description, it seems that you are running the deploy-tre.sh script from the bin/bash directory.

Try running it from the root directory of the project: ./bin/bash/deploy-tre.sh -r my-region

Closing this issue since it has not received a new comment recently. Please Open an issue if you are still blocked by this.

kaul-abhi commented 9 months ago

I am facing the same issue again Shashank/Scott @zillidan @smaly-amazon. I am deploying using your deploy-tre.sh script only like the way you mentioned and also go version is 1.20.7 but still error is coming.

smaly-amazon commented 9 months ago

Hi,

Are you sure you are running from the root directory of the repo, like this?

$ ./bin/bash/deploy-tre.sh  -r us-east-1
AWS account: xxxxxxxxxxxx
AWS region: us-east-1
Creating a Python virtual environment
Building the ServiceCatalogTerraformOSParameterParser function
go: creating new go.mod: module terraform_open_source_parameter_parser
go: to add module requirements and sums:
    go mod tidy
go: finding module for package github.com/aws/aws-sdk-go/aws/credentials
go: finding module for package github.com/hashicorp/hcl/v2/hclparse
go: finding module for package github.com/aws/aws-sdk-go/service/s3/s3manager
go: finding module for package github.com/stretchr/testify/mock
go: finding module for package github.com/aws/aws-sdk-go/aws/arn
go: finding module for package github.com/aws/aws-lambda-go/lambda
go: finding module for package github.com/aws/aws-sdk-go/aws/session
go: finding module for package github.com/aws/aws-sdk-go/aws/credentials/stscreds
go: finding module for package github.com/hashicorp/terraform-config-inspect/tfconfig
go: finding module for package github.com/aws/aws-sdk-go/service/s3
go: finding module for package github.com/aws/aws-sdk-go/aws
go: downloading github.com/hashicorp/hcl/v2 v2.19.1
go: downloading github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-lambda-go/lambda in github.com/aws/aws-lambda-go v1.41.0
go: found github.com/aws/aws-sdk-go/aws in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-sdk-go/aws/arn in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-sdk-go/aws/credentials in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-sdk-go/aws/credentials/stscreds in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-sdk-go/aws/session in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-sdk-go/service/s3 in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/aws/aws-sdk-go/service/s3/s3manager in github.com/aws/aws-sdk-go v1.47.5
go: found github.com/hashicorp/hcl/v2/hclparse in github.com/hashicorp/hcl/v2 v2.19.1
go: found github.com/hashicorp/terraform-config-inspect/tfconfig in github.com/hashicorp/terraform-config-inspect v0.0.0-20230925220900-5a6f8d18746d
go: found github.com/stretchr/testify/mock in github.com/stretchr/testify v1.8.4
go: downloading golang.org/x/net v0.17.0
go: downloading golang.org/x/text v0.13.0
Building the Lambda code
Collecting wheel
  Downloading wheel-0.41.3-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 KB 414.1 kB/s eta 0:00:00
Installing collected packages: wheel
Successfully installed wheel-0.41.3
WARNING: You are using pip version 22.0.4; however, version 23.3.1 is available.
You should consider upgrading via the '/Users/smaly/repos/service-catalog-engine-for-terraform-os/venv/bin/python3 -m pip install --upgrade pip' command.
Collecting boto3>=1.26.105
  Downloading boto3-1.28.80-py3-none-any.whl (135 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.8/135.8 KB 578.0 kB/s eta 0:00:00
Collecting botocore>=1.29.105
  Downloading botocore-1.31.80-py3-none-any.whl (11.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 26.8 MB/s eta 0:00:00
Collecting s3transfer<0.8.0,>=0.7.0
  Using cached s3transfer-0.7.0-py3-none-any.whl (79 kB)
Collecting jmespath<2.0.0,>=0.7.1
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
  Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.8/143.8 KB 5.6 MB/s eta 0:00:00
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.28.80 botocore-1.31.80 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.7.0 six-1.16.0 urllib3-1.26.18
WARNING: You are using pip version 22.0.4; however, version 23.3.1 is available.
You should consider upgrading via the '/Users/smaly/repos/service-catalog-engine-for-terraform-os/venv/bin/python3 -m pip install --upgrade pip' command.
Building codeuri:                                                               
/Users/smaly/repos/service-catalog-engine-for-terraform-os/lambda-functions/prov
isioning-operations-handler runtime: python3.9 metadata: {} architecture: x86_64
functions: TerraformEngineProvisioningHandlerLambda,                            
TerraformEngineTerminateHandlerLambda                                           
 Running PythonPipBuilder:ResolveDependencies                                   
 Running PythonPipBuilder:CopySource                                            
Building codeuri:                                                               
/Users/smaly/repos/service-catalog-engine-for-terraform-os/lambda-functions/terr
aform_open_source_parameter_parser runtime: go1.x metadata: {} architecture:    
x86_64 functions: ParameterParser, ExternalParameterParser                      
 Running GoModulesBuilder:Build                                                 
Building codeuri:                                                               
/Users/smaly/repos/service-catalog-engine-for-terraform-os/lambda-functions/stat
e_machine_lambdas runtime: python3.9 metadata: {} architecture: x86_64          
functions: SelectWorkerHostFunction, GetStateFileOutputsFunction,               
PollCommandInvocationFunction, SendApplyCommandFunction,                        
NotifyProvisionResultFunction, NotifyUpdateResultFunction,                      
SendDestroyCommandFunction, NotifyTerminateResultFunction                       
 Running PythonPipBuilder:ResolveDependencies                                   
 Running PythonPipBuilder:CopySource                                            

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided

SAM CLI update available (1.100.0); (1.97.0 installed)
To download: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html
Deploying the bootstrap bucket stack.
... etc ...

I did the above after pulling the latest commit from main and running git clean. Please let us know if we are missing something, but I think my previous answer is probably what you need to check.

kaul-abhi commented 9 months ago

Hi @smaly-amazon scott, thanks for your time on looking into my issue. I am actually running the deploy-tre.sh script from root only, I created shell script to clone this repo and run the deploy-tre.sh but no luck. PFB the script.

##################################################################

!/usr/bin/env bash

set -e

TRE_REPO_LOCATION="https://github.com/aws-samples/service-catalog-engine-for-terraform-os.git" echo "Clone the Terraform Reference Engine code from github." echo

git clone "https://gitlab-ci-token:$CI_JOB_TOKEN@$(echo $TRE_REPO_LOCATION | sed 's/https\?:\/\///')" TRE_DIRECTORY=$(basename -s .git $TRE_REPO_LOCATION | sed 's/.*\///')

log_info_command "cleaning template.yml file so to copy customized template.yaml" rm -rf $TRE_DIRECTORY/template.yaml

echo "copying customized template.yml file into TRE directory" cp /pipeline/template.yaml $TRE_DIRECTORY/

echo "Debugging path of destination directory" ls -lrt $TRE_DIRECTORY chmod -R 755 $TRE_DIRECTORY

cd $TRE_DIRECTORY

echo aws sts get-caller-identity echo

echo "running the deploy-tre script" ./bin/bash/deploy-tre.sh -r eu-west-1

#############################################

smaly-amazon commented 9 months ago

Hi,

Okay, I agree that it doesn't seem to be that you are running from the wrong directory.

I was curious and did a basic repro on my machine, like this.

#!/usr/bin/env bash
set -e

TRE_REPO_LOCATION="git@github.com:aws-samples/service-catalog-engine-for-terraform-os.git"
echo "Clone the Terraform Reference Engine code from github."
echo

#git clone "https://gitlab-ci-token:$CI_JOB_TOKEN@$(echo $TRE_REPO_LOCATION | sed 's/https?:////')"

git clone $TRE_REPO_LOCATION

#TRE_DIRECTORY=$(basename -s .git $TRE_REPO_LOCATION | sed 's/.*///')
TRE_DIRECTORY=service-catalog-engine-for-terraform-os

#log_info_command "cleaning template.yml file so to copy customized template.yaml"
#rm -rf $TRE_DIRECTORY/template.yaml
rm -f $TRE_DIRECTORY/template.yaml

echo "copying customized template.yml file into TRE directory"
#cp /pipeline/template.yaml $TRE_DIRECTORY/

# Note here that /tmp/pipeline/template.yaml is identical to what is in the repo.
cp /tmp/pipeline/template.yaml $TRE_DIRECTORY/

echo "Debugging path of destination directory"
ls -lrt $TRE_DIRECTORY
chmod -R 755 $TRE_DIRECTORY
cd $TRE_DIRECTORY
echo
aws sts get-caller-identity
echo
echo "running the deploy-tre script"

#./bin/bash/deploy-tre.sh -r eu-west-1
./bin/bash/deploy-tre.sh -r us-east-1

This worked fine.

The one difference here is your custom template.yaml file. Have you changed the Handler property for the ParameterParser or ExternalParameterParser resources?

Sorry this has been a bit of a mystery for you.

kaul-abhi commented 9 months ago

@smaly-amazon No i didnt changed anything in the handler part. Also as part of debugging i also removed my customized template and tried to install without it but no luck. My base image is alpine actually, not sure if its due to some go package that is missing. I tried with go version 1.21.0 but still no luck.