Unable to perform cdktf diff when using multiple stacks with inter-stack dependencies

natemellendorf commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

language: python cdktf-cli: 0.12.2 node: v18.10.0 cdktf: 0.12.3 constructs: 10.1.119 jsii: 1.69.0 terraform: 1.3.0 arch: arm64 os: linux 5.10.104-linuxkit python: Python 3.9.13 pip: pip 22.2.2 from /root/.local/share/virtualenvs/infra-o81V4Mez/lib/python3.9/site-packages/pip (python 3.9) pipenv: pipenv, version 2022.9.24

Affected Resource(s)

Any/All

Debug Output

The following is returned, when attempting to run a plan against multiple stacks:

cdktf plan '*-stack'
...
Usage Error: Could not find stack: *-stack

cdktf plan first-stack cdktf plan second-stack

Expected Behavior

Similar to cdktf apply, I believe I should be able to run *cdktf plan '-stack'** and review what would happen, should the matched stacks be applied.

Actual Behavior

The following error is returned: Usage Error: Could not find stack: *-stack

Steps to Reproduce

Using Python and the CDKTF, and these two stacks, attempt to run cdktf plan '*-stack'

Important Factoids

You can plan first-stack just fine with: cdktf plan first-stack

However, if you try and plan the second stack (having not actually deployed first-stack), I get the following error:

second-stack  data.terraform_remote_state.cross-stack-reference-input-first-stack: Reading...

⠼  Processing
[2022-10-02T19:31:38.681] [ERROR] default - ╷
│ Error: Unable to find remote state
│ 
│   with data.terraform_remote_state.cross-stack-reference-input-first-stack,
│   on cdk.tf.json line 18, in data.terraform_remote_state.cross-stack-reference-input-first-stack:
│   18:         "workspace": "${terraform.workspace}"
│ 
│ No stored state was found for the given workspace in the given backend.
second-stack  ╷
              │ Error: Unable to find remote state
              │ 
              │   with data.terraform_remote_state.cross-stack-reference-input-first-stack,
              │   on cdk.tf.json line 18, in data.terraform_remote_state.cross-stack-reference-input-first-stack:
              │   18:         "workspace": "${terraform.workspace}"
              │ 
              │ No stored state was found for the given workspace in the given backend.
              ╵

⠴  Processing

The above behavior makes sense. Second-stack fails because it can't find a data resource that it expects from first-stack. However, with that behavior, I do not see a way to plan multiple stacks that have inter-stack dependencies.

cdktf apply and cdktf destroy seem to work just fine.

References

ste-wr commented 2 years ago

Yep this is affecting me as well. This is pretty much a blocker for multi-stack in CI using cdktf commands unless we fall-back to some kind of recursive terraform plan after a cdktf synth

jsteinich commented 2 years ago

While not impossible, supporting this in all cases would be a large departure from how diff currently works. Cross-stack references are accomplished through the use of Terraform remote state data sources. These rely on the remote state (1st stack) existing (already deployed). When that is the case, the diff will work just fine. When that isn't the case is where it gets tricky.

A rough outline of how this could work:

Diff (plan) the 1st stack
Resyth/fill in the 2nd stack using the results of that plan
Diff the 2nd stack

ste-wr commented 2 years ago

@jsteinich That's how I assumed it would work.

Are you suggesting that diff should work fine at the moment for multiple stacks where there's no dependency? Because my experience (with 0.13.0 at least) is that this isn't the case. In all cases, I get the Could not find stack/specify a stack error if I try to run vanilla cdktf diff.

Maybe this is a docs improvement, because when I read that CDKTF supports multi-stack deployment, I kinda expect that the entire core command set (diff, synth, deploy, destroy) would support it.

natemellendorf commented 2 years ago

I think we all understand the issue at hand, which is the way dynamic data blocks are created for stacks with dependencies.

So the CDK does this automatically. When you create two stacks, one that uses the output of another, the terraform synthesized will create the second stack with a dynamically named remote state data block.

In theory, it makes sense. In practice, it falls short. This is due to how a data block can’t be planned if the resource it represents does not exist.

Per this discussion, I don’t think there’s any plan to address that behavior. https://github.com/hashicorp/terraform/issues/16380

At this point, I think this is in the hands of the CDKTF/TF team to solve. Either allow data blocks to return empty (so plans can succeed) or rework how CDKTF implements shared resources between two stacks.

If neither, then please communicate to the community that using CDKTF with multiple stacks is quite limited.

jsteinich commented 2 years ago

Are you suggesting that diff should work fine at the moment for multiple stacks where there's no dependency?

No. Apologizes if I gave that impression. I simply meant that it would be a simple addition for this case.

At this point, I think this is in the hands of the CDKTF/TF team to solve. Either allow data blocks to return empty (so plans can succeed) or rework how CDKTF implements shared resources between two stacks.

I wouldn't expect this to change in Terraform directly (judging by their responses). The CDKTF team applied a label (needs-priority) which means they are looking for community input to determine whether or not they will work on this. Even if they decide it's a lower priority, someone from the community is always welcome to contribute towards this.

elovelan commented 11 months ago

Will Terraform's native support for stacks help move this forward?

If so, can the community (myself included) help with implementation?

hashicorp / terraform-cdk