DABS Deploy not producing or not able to find Terraform State file

[FreeYourData]

Hi, We are using DABS to deploy our Python notebooks as a bundle.

Issue:

databricks bundle deploy command is not producing the Terraform STATE file (.tfstate) or nowhere to be found.

Background:

Due to Firewall restrictions we are forced to setup DB CLI in Isolated (airgapped) mode. We have downloaded Terraform binary, Databricks TF provider, configured all necessary System Environment Variables in a Windows OS. Within our Development VDI we are now able to validate and deploy our Bundle.

Also, we have configured a DevOps Pipeline to perform CI/CD with DB CLI. Within the DABS YAML file we have specified the location for the State file (state_path).

However we are still not able to generate the .tfstate file neither locate it (locally or in the target Workspace. We know that Terraform requires it in order to generate a plan.

Configuration

• Terraform compiler (binary) 1.9.8 Windows AMD64
• Databricks Terraform provider 1.54.0 Windows AMD64.

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Define in DABS YAML the location of the Workspace root_path, State_path and artifacts_path
2. Run databricks bundle deploy -p <profile-name> -t <target-name>
3. See error

Expected Behavior

We expected to also see the Terraform State file (.tfstate)

Actual Behavior

The .tfstate file is only locally

OS and CLI version

• Databricks CLI v0.233.0.

Is this a regression?

Debug Logs

Streaming response pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire sdk=true 07:56:16 DEBUG Apply pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred 07:56:16 DEBUG Apply pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq 07:56:16 DEBUG Apply pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull 07:56:16 DEBUG GET /api/2.0/workspace/get-status?path=/Workspace//< username >/.bundle/< bundle name >/< target name >/state/terraform.tfstate&return_export_info=true HTTP/2.0 404 Not Found { "error_code": "RESOURCE_DOES_NOT_EXIST", "message": "Path (/Workspace/Users/< username >/.bundle/< bundle name >/< target name >/... (52 more bytes)" } pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true 07:56:16 DEBUG non-retriable error: Path (/Workspace/Users/< username >/.bundle/< bundle name >/< target name >/state/terraform.tfstate) doesn't exist. pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true 07:56:16 INFO Remote state file does not exist. Using local Terraform state. pid=4772 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull

[andrewnester]

@FreeYourData could you provide the full debug logs and your bundle configuration? The logs you provided does not indicate the error and are actually expected for the first bundle deployment because state file indeed does not exist remotely yet.

[FreeYourData]

@andrewnester Do you have a safe location to share it? Company policy as the log may contain some sensitive information. Or maybe better if I remove it, not sure what is the best way to proceed here.

[andrewnester]

Please remove any sensitive information before sharing the logs. We don't need the full logs, just what is the actual error you receive when you run "databricks bundle deploy"?

[FreeYourData]

dabs_log.txt I've removed all possible sensitive data. Thanks in advanced

[andrewnester]

@FreeYourData thanks! what is the value of your state location path in your bundle configuration?

[FreeYourData]

@andrewnester For now not much besides testing in order to find out where the state file is to be found. We expect in the future to keep each state file per bundle (project in our terms)

[andrewnester]

@FreeYourData can you share your bundle configuration?

[FreeYourData]

DABS Configuration.txt I removed any possible sensiteve data and saved it as txt but you can redefine it as YAML

[andrewnester]

@FreeYourData you don't have any resources like job, pipelines and etc defined in your bundle hence no state is generated. As soon as you define any resources the state will be generated and uploaded.

Also you don't necessarily need to include all files explitictly, they are synced automatically on bundle deploy so you can remove this section

  include:
    - main_validation.py
    - pre_validation.py
    - utilities_validation.py
    - metadata_validation.py

[FreeYourData]

@andrewnester this is great input, thanks a lot ! No wonder we could not find it. As we where initially working with the python template that had jobs and pipelines therefore we saw the state file, after we moved to manual DABS creation we missed it. Also thanks for the extra suggestion. Much appreciated!