microsoft / azure-pipelines-terraform

Azure Pipelines tasks for installing Terraform and running Terraform commands in a build or release pipeline.
MIT License
95 stars 59 forks source link

AWS S3 terraform init remote backend task successful, but dont find terraform state in s3 bucket #180

Open onenessboy opened 10 months ago

onenessboy commented 10 months ago

Hi,

I am using azure devops classic pipeline for deploying infrastructure onto AWS, Steps I have followed

First created two Terraform for AWS plugin service connection with an access key and secret of an IAM user who had access as administrator into my target AWS account

I have replace token,terraform install, terraform init, terraform plan tasks where in terraform Init, I have configure amazon backend connection with which i created above...

`C:\hostedtoolcache\windows\terraform\1.5.6\x64\terraform.exe init -backend-config=bucket=iac-terraform -backend-config=key=tf/terraform.tfstate -backend-config=region= -backend-config=access_key= -backend-config=secret_key=***

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes.

Initializing provider plugins...

struggling with this issue since last two days

1) so even if you define backend config in your provider main.tf, azure devops task value over rides it correct ? or we dont need to provide backed definition in proviermain. tf as pipele terraform task already have backend config info since I am using classical pipeline... ? 2) the iam user which I have used have admin access and full access on s3 3) while i am this service connection, all input we give is access key, secret key and region, that means, based on access key and secret key it will determine which aws accounts it will connect ? since IAM user is created at global level, so s3 bucket also global level right...

really not sure , task is successful but I dont see any terraform state file there ? how to debug this any idea...

mericstam commented 8 months ago

Hi, Sorry for late reply. please run the build with debug flag checked and submit the log after clearing out sensitive information

onenessboy commented 8 months ago

Hi, Sorry for late reply. please run the build with debug flag checked and submit the log after clearing out sensitive information

@mericstam Thanks for your reply, Here is my debug log of Init command

2023-10-21T15:31:36.8094886Z ##[debug]Evaluating condition for step: 'Terraform : aws: Init' 2023-10-21T15:31:36.8096875Z ##[debug]Evaluating: succeeded() 2023-10-21T15:31:36.8097392Z ##[debug]Evaluating succeeded: 2023-10-21T15:31:36.8098651Z ##[debug]=> True 2023-10-21T15:31:36.8099106Z ##[debug]Result: True 2023-10-21T15:31:36.8099631Z ##[section]Starting: Terraform : aws: Init 2023-10-21T15:31:36.8237702Z ============================================================================== 2023-10-21T15:31:36.8238129Z Task : Terraform 2023-10-21T15:31:36.8238290Z Description : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP) 2023-10-21T15:31:36.8239176Z Version : 3.209.24 2023-10-21T15:31:36.8239340Z Author : Microsoft Corporation 2023-10-21T15:31:36.8239588Z Help : Learn more about this task 2023-10-21T15:31:36.8239965Z ============================================================================== 2023-10-21T15:31:36.8902141Z ##[debug]Using node path: C:\agents\3.227.2\externals\node10\bin\node.exe 2023-10-21T15:31:37.9920701Z ##[debug]agent.TempDirectory=D:\a_temp 2023-10-21T15:31:37.9954155Z ##[debug]loading inputs and endpoints 2023-10-21T15:31:37.9967920Z ##[debug]loading ENDPOINT_AUTH_xxxxxxxxxxxxxxxx 2023-10-21T15:31:38.3531588Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_xxxxxxxxxxxxxxx_PASSWORD 2023-10-21T15:31:38.3534614Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_xxxxxxxxxxxxxxxxxxxx_REGION 2023-10-21T15:31:38.3537203Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_xxxxxxxxxxxxxxx_USERNAME 2023-10-21T15:31:38.3540022Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN 2023-10-21T15:31:38.3541637Z ##[debug]loading ENDPOINT_AUTH_SCHEME_xxxxxxxxxxxxxxx 2023-10-21T15:31:38.3543882Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION 2023-10-21T15:31:38.3546042Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION 2023-10-21T15:31:38.3549454Z ##[debug]loading INPUT_BACKENDAWSBUCKETNAME 2023-10-21T15:31:38.3551732Z ##[debug]loading INPUT_BACKENDAWSKEY 2023-10-21T15:31:38.3554654Z ##[debug]loading INPUT_BACKENDSERVICEAWS 2023-10-21T15:31:38.3556925Z ##[debug]loading INPUT_COMMAND 2023-10-21T15:31:38.3559783Z ##[debug]loading INPUT_OUTPUTFORMAT 2023-10-21T15:31:38.3562091Z ##[debug]loading INPUT_OUTPUTTO 2023-10-21T15:31:38.3564642Z ##[debug]loading INPUT_PROVIDER 2023-10-21T15:31:38.3566717Z ##[debug]loading INPUT_WORKINGDIRECTORY 2023-10-21T15:31:38.3576785Z ##[debug]loading SECRET_AWS_ACCESS_KEY 2023-10-21T15:31:38.3579475Z ##[debug]loading SECRET_AWS_SECRET_KEY 2023-10-21T15:31:38.3595667Z ##[debug]loaded 18 2023-10-21T15:31:38.3604239Z ##[debug]Agent.ProxyUrl=undefined 2023-10-21T15:31:38.3605192Z ##[debug]Agent.CAInfo=undefined 2023-10-21T15:31:38.3606292Z ##[debug]Agent.ClientCert=undefined 2023-10-21T15:31:38.3609212Z ##[debug]Agent.SkipCertValidation=undefined 2023-10-21T15:31:38.4775078Z ##[debug]check path : D:\a_tasks\TerraformTaskV3_fe504acc-6115-40cb-89ff-191386b5e7bf\3.209.24\task.json 2023-10-21T15:31:38.4780409Z ##[debug]adding resource file: D:\a_tasks\TerraformTaskV3_fe504acc-6115-40cb-89ff-191386b5e7bf\3.209.24\task.json 2023-10-21T15:31:38.4781757Z ##[debug]system.culture=en-US 2023-10-21T15:31:38.4810942Z ##[debug]provider=aws 2023-10-21T15:31:38.4812759Z ##[debug]command=init 2023-10-21T15:31:38.4821697Z ##[debug]workingDirectory=D:\a\1\s\Terraform 2023-10-21T15:31:38.4822666Z ##[debug]commandOptions=undefined 2023-10-21T15:31:38.4824374Z ##[debug]which 'terraform' 2023-10-21T15:31:38.4843265Z ##[debug]found: 'C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe' 2023-10-21T15:31:38.4844681Z ##[debug]which 'C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe' 2023-10-21T15:31:38.4846259Z ##[debug]found: 'C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe' 2023-10-21T15:31:38.4847998Z ##[debug]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe arg: init 2023-10-21T15:31:38.4851632Z ##[debug]backendServiceAWS=xxxxxxxxxxxxxxx 2023-10-21T15:31:38.4854024Z ##[debug]backendAWSBucketName=dev-tfstate 2023-10-21T15:31:38.4855380Z ##[debug]backendAWSKey=tf/terraform.tfstate 2023-10-21T15:31:38.4858447Z ##[debug]xxxxxxxxxxxxxxx auth param region = 2023-10-21T15:31:38.4859752Z ##[debug]xxxxxxxxxxxxxxx auth param username = 2023-10-21T15:31:38.4861337Z ##[debug]xxxxxxxxxxxxxxx auth param password = 2023-10-21T15:31:38.4864767Z ##[debug]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe arg: -backend-config=bucket=dev-tfstate 2023-10-21T15:31:38.4866527Z ##[debug]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe arg: -backend-config=key=tf/terraform.tfstate 2023-10-21T15:31:38.4868263Z ##[debug]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe arg: -backend-config=region= 2023-10-21T15:31:38.4870056Z ##[debug]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe arg: -backend-config=access_key= 2023-10-21T15:31:38.4871944Z ##[debug]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe arg: -backend-config=secret_key= 2023-10-21T15:31:38.4873417Z ##[debug]exec tool: C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe 2023-10-21T15:31:38.4874393Z ##[debug]arguments: 2023-10-21T15:31:38.4874966Z ##[debug] init 2023-10-21T15:31:38.4875819Z ##[debug] -backend-config=bucket=dev-tfstate 2023-10-21T15:31:38.4876691Z ##[debug] -backend-config=key=tf/terraform.tfstate 2023-10-21T15:31:38.4877537Z ##[debug] -backend-config=region= 2023-10-21T15:31:38.4878366Z ##[debug] -backend-config=access_key= 2023-10-21T15:31:38.4879345Z ##[debug] -backend-config=secret_key= 2023-10-21T15:31:38.4885323Z [command]C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe init -backend-config=bucket=dev-tfstate -backend-config=key=tf/terraform.tfstate -backend-config=region= -backend-config=access_key= -backend-config=secret_key= 2023-10-21T15:31:38.6066499Z 2023-10-21T15:31:38.6067616Z Initializing the backend... 2023-10-21T15:31:38.6106021Z 2023-10-21T15:31:38.6107164Z Initializing provider plugins... 2023-10-21T15:31:38.6108008Z - Finding latest version of hashicorp/local... 2023-10-21T15:31:38.8112573Z - Finding latest version of hashicorp/template... 2023-10-21T15:31:38.8524199Z - Finding hashicorp/aws versions matching "~> 3.0"... 2023-10-21T15:31:38.8912536Z - Finding latest version of hashicorp/tls... 2023-10-21T15:31:39.0056378Z - Installing hashicorp/local v2.4.0... 2023-10-21T15:31:39.3503387Z - Installed hashicorp/local v2.4.0 (signed by HashiCorp) 2023-10-21T15:31:39.4371174Z - Installing hashicorp/template v2.2.0... 2023-10-21T15:31:39.9308733Z - Installed hashicorp/template v2.2.0 (signed by HashiCorp) 2023-10-21T15:31:40.0269658Z - Installing hashicorp/aws v3.76.1... 2023-10-21T15:31:43.9029111Z - Installed hashicorp/aws v3.76.1 (signed by HashiCorp) 2023-10-21T15:31:43.9891961Z - Installing hashicorp/tls v4.0.4... 2023-10-21T15:31:44.3831643Z - Installed hashicorp/tls v4.0.4 (signed by HashiCorp) 2023-10-21T15:31:44.3832639Z 2023-10-21T15:31:44.3834519Z Terraform has created a lock file .terraform.lock.hcl to record the provider 2023-10-21T15:31:44.3835612Z selections it made above. Include this file in your version control repository 2023-10-21T15:31:44.3861833Z so that Terraform can guarantee to make the same selections by default when 2023-10-21T15:31:44.3891048Z you run "terraform init" in the future.

2023-10-21T15:31:44.3940315Z Terraform has been successfully initialized! 2023-10-21T15:31:44.3940982Z  2023-10-21T15:31:44.3941490Z You may now begin working with Terraform. Try running "terraform plan" to see 2023-10-21T15:31:44.3942432Z any changes that are required for your infrastructure. All Terraform commands 2023-10-21T15:31:44.3943329Z should now work. 2023-10-21T15:31:44.3943515Z 2023-10-21T15:31:44.3943976Z If you ever set or change modules or backend configuration for Terraform, 2023-10-21T15:31:44.3944899Z rerun this command to reinitialize your working directory. If you forget, other 2023-10-21T15:31:44.3945776Z commands will detect it and remind you to do so if necessary. 2023-10-21T15:31:44.4024150Z ##[debug]Exit code 0 received from tool 'C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe' 2023-10-21T15:31:44.4037006Z ##[debug]STDIO streams have closed for tool 'C:\hostedtoolcache\windows\terraform\1.6.2\x64\terraform.exe' 2023-10-21T15:31:44.4038333Z ##[debug]task result: Succeeded 2023-10-21T15:31:44.4040407Z ##[debug]Processed: ##vso[task.complete result=Succeeded;] 2023-10-21T15:31:44.4715219Z ##[section]Finishing: Terraform : aws: Init

Task is successful but I dont see it created terraform.tf state file on s3 bucket