Pulumi stack name as compiler argument

[artem-zherdiev-ingio]

Hi, thank you for amazing tool. I am trying to customize pulumi yaml for our needs, and thinking about usage Pulumi.<stack>.yaml as source of our resources/outputs and compiler option with our custom go program which will generate pulumi.yaml and under the hood will have some features like go/template package, templates of our common resources, etc.

It seems currently is no way to give compiler knowing which stack it using. Could Pulumi.yaml has something like:

options:
   compiler: ./compiler "${pulumi.stack}"

Thank you for response!

[Frassle]

"compiler" doesn't currently support variable interpolation, but I think we could add the built in variables here. Small backwards compatibility concern here that people might already have command line arguments using ${}.

[artem-zherdiev-ingio]

@Frassle Hi, just to understand when this might be implemented or if it will definitely be implemented someday, so we can create our workaround now to take it from env until it will from argument. Thank you!

[Frassle]

OK I looked into this a bit more. Adding something like this would be problematic. For convert and for getting the plugins required because those also needs to load the template (with the compiler if set), but there is no stack name in those scenarios.

Now might still be we add something like this but I think we'd have to mark all variables as empty when doing plugin query and conversions, so we'd call your example with ./compiler "" and it would have to handle that.

That sounds unlikely to work in general, if you can compile the template without the stack name then you should just do that all the time, and if you can't correctly generate the template without the stack name than plugin queries and conversions would be wrong.

I'm inclined to think that this is probably the wrong way to solve the problem your trying to address.

[artem-zherdiev-ingio]

Well, maybe we need more context. We want to switch from pulumi typescript to pulumi yaml. We manage several clusters using pulumi stacks. Some places are similar and we can use configs, some places have additional resources, some places we shouldn't use some resources. So,

1. we need conditions, which, as far as I know, pulumi yaml doesn't support yet,
2. we want to create templates for each of our resources to reuse with some changes for each stack. We want to store it in our folder structure and manage it inside pulumi.<stack>.yaml,
3. It would be great to be able to create loops when needed, which we can achieve also with golang text/template.

The ability to create your own compiler is a great feature. We can do whatever we want with yaml before outputting it the way pulumi requires. But how do we deal with all this if we have separate stacks and the compiler doesn't know which stack pulumi is using? For me, it doesn't matter how the compiler can get the stack name, it can be from an argument, from the env key if pulumi will process this key at runtime, or even from a file if it is automatically generated. Ultimately, I believe this feature allows the compiler to be more flexible.

If you think this is the wrong way to handle such needs, perhaps I don't know the current functionality of pulumi yaml deeply enough and it is somehow possible. I would appreciate your help and further discussion. Thanks

[Frassle]

I think your trying to use multiple stacks but what you actually want are multiple projects.

If you split this into multiple projects each Pulumi.yaml could call into your compiler with arguments to tell it what sort of program to build. That still means each project can fully build it's yaml template for our yaml runtime engine to be able to answer queries from the engine for what plugins are needed, or to convert out to other languages.

[artem-zherdiev-ingio]

I don't think so, we have dev, staging, prod, infra stacks for gcloud. From my perspective, to separate it into projects they should have much more differences. For example, if we have gcloud and aws, yes it should be in different projects.

So, I am right understand that pulumi yaml currently does handle the stack concept only for small config changes, if you want something bigger it should be separated it all into projects? And more importantly, the idea of getting the compiler to know which stack is using (anyway how) it's not possible?

[Frassle]

So, I am right understand that pulumi yaml currently does handle the stack concept only for small config changes, if you want something bigger it should be separated it all into projects?

Yeh this is because of the lack of conditions in the final yaml. It's ok to have dev/staging/prod as long as they differ by things like "how big the VM to build is", but not if they differ by "should a VPC be created or not". I think this is a fair case to re-visit that design and maybe allow conditions here. https://github.com/pulumi/pulumi-yaml/issues/191 is currently tracking that idea.

And more importantly, the idea of getting the compiler to know which stack is using (anyway how) it's not possible?

That's probably never going to happen. We need to be able to generate the final yaml for plugin queries and program conversions and there really is no stack name in those scenarios.

I think the overall answer here is this ticket isn't going to happen, but to follow up in https://github.com/pulumi/pulumi-yaml/issues/191 to enable your use case of having conditions between different environments. If you've got some specific cases that you could share in that ticket it would help.

[artem-zherdiev-ingio]

The last try :) Does any possibility exist to hook pulumi stack select ... for storing stack name on our side? Maybe I will create own quick plugin for it or so.

By the way, IMHO make compiler much stronger is better idea that improve pulumi yaml like #191, at least you will give more power to devs. And pulumi was designed to use program languages for that ;)

[Frassle]

Does any possibility exist to hook pulumi stack select

You could just exec pulumi stack --show-name to get the current name, but it might return error: no stack selected; please use `pulumi stack select` or `pulumi stack init` to choose one

[artem-zherdiev-ingio]

Yes, I thought about it too.

Ok, if you say it will never going to happen that ticket may be closed for now. Thank you for quick responses.