Undeprecate spinnaker_pipeline resource/datasource

[andrein]

WHAT

Add test coverage and undeprecate spinnaker_pipeline resource and datasource. Willing to step up and maintain them for the foreseeable future.

WHY

From my understanding, the plan is to replace it with a more HCL-friendly resource. When/if the new resource type is implemented, there will still be a lot of code using the current resource with no straightforward migration path.

We're currently experiencing the migration pain from the other point of view: our workflow currently uses jgramoll/terraform-provider-spinnaker which does provide a HCL-friendly interface, but we're switching to sponnet + terraform-provider-jsonnet + mercari/terraform-provider-spinnaker to render our pipelines because the other provider is very slow and isn't as flexible as sponnet. There's also #71, which we will need to implement in order to migrate our pipelines, but I'd like to hear your thoughts/feedback before I start.

[KeisukeYamashita]

@andrein Nice, thank you for creating this issue.

Add test coverage and undeprecate spinnaker_pipeline resource and datasource. Willing to step up and maintain them for the foreseeable future.

This contains two parts, 1. adding test coverage and undeprecate spinnaker_pipeline resource and datasource.

I created a new issue here → https://github.com/mercari/terraform-provider-spinnaker/issues/94

From my understanding, the plan is to replace it with a more HCL-friendly resource

Yes, I'm planning to provide an API to describe the pipeline with HCL.

When/if the new resource type is implemented, there will still be a lot of code using the current resource with no straightforward migration path.

I think the migration path is not that hard. I will create a prototype and get feedback here by the end of 2020. I'm planing to implement like Datadog's Dashboard widget (Ref: https://github.com/DataDog/terraform-provider-datadog/blob/master/datadog/resource_datadog_dashboard.go). I don't want to support JSON inputs but want to support only HCL inputs.

[andrein]

Hi @KeisukeYamashita

When/if the new resource type is implemented, there will still be a lot of code using the current resource with no straightforward migration path.

I think the migration path is not that hard. I will create a prototype and get feedback here by the end of 2020. I'm planing to implement like Datadog's Dashboard widget (Ref: https://github.com/DataDog/terraform-provider-datadog/blob/master/datadog/resource_datadog_dashboard.go).

I did some thinking about a similar resource about a year ago, and the major issue I hit was that representing what is essentially a graph structure in HCL is very difficult. Think about pipelines that deploy something in a few different kubernetes clusters in parallel - we have been doing that since the days of the kubernetes v1 spinnaker provider and decided to keep doing it when we switched to v2.

jgramoll/terraform-provider-spinnaker makes it easy to create complex pipelines in HCL, since it implements each stage as a resource. However it it very slow, since it has to update the pipeline for every step. For large pipelines, this is very slow. For a project with multiple large pipelines, you have to lower parallelism and/or use resource targeting (at some point we had apply runs that took 20 minutes for a few dozen pipelines, although almost nothing changed).

Doing it in a single resource should make it fast, but we're back to representing a complex graph using HCL. I was thinking of using nested blocks to represent the stages, but that proved difficult since you can't reference a nested block from another. You can do it by assigning a unique identifier to every resource manually and build the graph that way, but I think this solution isn't very user friendly.

These are just some of my mental notes from a year ago, stuff might have changed in the terraform plugin sdk that would make building references between nested blocks easier.

I'm looking forward to your proof of concept, though, I'd be happy to take it for a spin.

I don't want to support JSON inputs but want to support only HCL inputs.

I'm a bit confused, should I take this as "the spinnaker_pipeline resource will stay deprecated and be removed at some point in the future" or as "the new HCL resource won't support JSON inputs"?

IMO, keeping the old resource supported will provide an excellent escape hatch for people that want full control over their pipeline definitions.

Thanks, Andrei

[bilenkis]

+1 for keeping the possibility to use the externally rendered JSON object and use it as a source for spinnaker_pipeline. We're doing pretty much the same with the tidal-engineering provider but it doesn't support a lot of other entities, like canary configs.

[silvez-dh]

+1 for the above

[Jurawa]

Have to echo the request to keep the JSON pipeline source, as we are currently migrating to your provider from the outdated armory version, and we use jsonnet to generate our pipeline JSON to pass into the pipeline resource - which is why we chose this provider over the other HCL based ones.

Perhaps HCL could be a separate resource or argument to spinnaker_pipeline, but with the option to still use JSON?

Also, thanks for your work on this provider!