Inconsistent UX for multi-stack cdktf commands

daniel-laszlo commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

cdktf 0.13.2

Affected Resource(s)

cdktf synth cdktf diff cdktf deploy

Debug Output

Expected Behavior

Can we standardize options for handling multi-stack workflows? I need to be able to run all subcommands for 1) a specific stack, 2) for all stacks.

Actual Behavior

cdktf synth runs for all stacks without supplying any further options, but I cannot specify to run for only one stack
cdktf diff requires specifying a stack, but there is no option to run for all stacks
cdktf deploy requires specifying a stack or '*' to run for all stacks

Steps to Reproduce

Important Factoids

References

DanielMSchmidt commented 2 years ago

Yeah this part is a bit confusing and we don't really call the reason it explicitly in our documentation, so we should definitely add a section.

Synth synthesizes all stacks because synthesizing means running the app command. The command is arbitrary, but most commonly we run ts /python/ etc to execute your cdktf program. We can't run half a program that easily, so that's why we run it entirely.

When we built the multi-stack feature we wanted to allow to plan multiple stacks as well, but that's not really feasible right now. With cross stack references you might use a value from one stack in another. If we didn't apply yet or the changes would change that referenced value we can not make a (correct) plan. This is why we limit the plan to one stack.

daniel-laszlo commented 2 years ago

Thank you for the explanation

daniel-laszlo commented 1 year ago

@DanielMSchmidt followup question. Would I lose the ability to run plan for a stack which references a value from another stack? And then I would be restricted to only deploy in this case?

DanielMSchmidt commented 1 year ago

No, you can plan the stack housing the value, apply it and the plan the stack using the value. If you plan the stack using the value first your plan might be incorrect / erroring

LyleDavis commented 1 year ago

@DanielMSchmidt in the scenario of running a plan on a stack where it depends on another stack that hasn't yet had the relevant changes applied, is it guaranteed that it will error or will it sometimes show an incorrect plan without erroring?

I'm new to CDKTF and terraform in general, but in CI I was hoping to be able to run a diff on all stacks (if not together then one by one) and then deploy * on merge. Obviously that would be dangerous if there is the potential for the diff to be wrong due to stack dependencies.

fathom-parth commented 1 year ago

@DanielMSchmidt a tiny bump if you have time to answer @LyleDavis's question! I'm setting up cdktf for my organization and trying to make sure our CI/CD setup is optimal.

I think given this issue, the issue at https://github.com/hashicorp/terraform-cdk/issues/2157, and the cdktf best practices telling us to use multiple stacks over a single stack per-environment/project (we use gcp), there's no real way to run cdktf diff during CI?

jsteinich commented 1 year ago

in the scenario of running a plan on a stack where it depends on another stack that hasn't yet had the relevant changes applied, is it guaranteed that it will error or will it sometimes show an incorrect plan without erroring?

Both situations are possible. If the value doesn't exist at all an error will occur when performing a diff; however, if the value is being changed, the existing value would be used and lead to an incorrect plan.

I think given this issue, the issue at #2157, and the cdktf best practices telling us to use multiple stacks over a single stack per-environment/project (we use gcp), there's no real way to run cdktf diff during CI?

It really depends how the stacks are being used. Separate environments that don't reference each other will diff without issue. Dependent stacks can certainly cause diff issues, but it is still possible to use with some extra coordination.

fathom-parth commented 1 year ago

@jsteinich

It really depends how the stacks are being used. Separate environments that don't reference each other will diff without issue.

So I think the main issue is that there's no way that I can see to cdktf diff all the "leaf" stacks (i.e. the stacks with no dependencies) programmatically easily.

Without this, if a user adds a stack, they'll have to change the diff job explicitly to include their new stack. But our deploy job just runs cdktf deploy '*'. Ideally a user wouldn't have to worry about changing either job.

Dependent stacks can certainly cause diff issues, but it is still possible to use with some extra coordination.

In our case, the issue is that many stacks would be dependent on say the networking stack so that they can all use the same network. In this case the networking stack would rarely change and I still want to see the diff output for all the other stacks.

You mention "it is still possible to use with some extra coordination", could you expand a little on this?

jsteinich commented 1 year ago

There is a stack manifest created that includes dependency information. It's possible to write a script that reads it in and then diffs in an appropriate order. Would probably also need to be multi-step so that dependencies can be resolved (or just error if a potential issue is detected).

fathom-parth commented 1 year ago

Ah so we were looking into that but we'd still have to call cdktf deploy on (using the previous example again) the networking stack before being able to diff the downstream stacks?

It doesn't seem like a great practice to call deploy before merging the PR.

jsteinich commented 1 year ago

Ah so we were looking into that but we'd still have to call cdktf deploy on (using the previous example again) the networking stack before being able to diff the downstream stacks?

It doesn't seem like a great practice to call deploy before merging the PR.

A workaround for the initial creation could be to have a conditional for creating the downstream stack(s) that is disabled for the 1st PR and a 2nd PR enables them. Updates to the networking stack that need to ripple doesn't seem to have a great workaround.

I certainly agree with all the issues here. Just looking for workarounds since a generic solution would be a complex addition to cdktf.

fathom-parth commented 1 year ago

Ooh! I like that!

Is there a way to add a resource to the cross-stack outputs without actually having that resource be used cross-stack yet?

Then we could make a PR to the networking stack that exposes some new resource as a cross-stack resource without any other stack actually using that output yet. This would allow all diff calls to still work!

Then we could make a 2nd PR to the downstream stack that wants to use the new resource in the networking stack and the diff will work because the remote state will already have the cross-stack reference!

jsteinich commented 1 year ago

Is there a way to add a resource to the cross-stack outputs without actually having that resource be used cross-stack yet?

Yes, create a TerraformOutput containing the data you want to use. Cross-stack references make use of Terraform remote state data sources. In order for something to be available as such, it must be put into an output. If you name it the same as will be generated I believe it will link up. If that fails, you can also explicitly reference as remote state.

fathom-parth commented 1 year ago

Ideally we can name it the same so that we can pass the reference normally in the 2nd PR without having to define a TerraformOutput -- will test!

hashicorp / terraform-cdk