RFE: Use terraform to access multiple provider apis from a single library

[cpu]

Suggested by @marafa in https://github.com/jlund/streisand/issues/469

https://www.terraform.io/intro/index.html / https://github.com/hashicorp/terraform

[nickolasclarke]

getting aliyun would be great, but I dont see many of the other commonly requested providers like vultr and others I cant remember off the top of my head. It would be a significant project to migrate parts / all of the codebase to terraform. I think we'd need more value out of that before considering this. What other advantages might using terraform bring?

[cpu]

I'm particularly adverse to adding new provider support to Streisand. It's a lot of work, the libraries/APIs for providers change/break frequently, and it adds a huge test burden. Azure support is a perfect example: it was merged but 99% of the time the active maintainers don't test against it because none of us are Azure users. It's almost certainly going to break if it hasn't already.

For Streisand vs other cloud deployment scenarios I think the work of creating the initial instance is low and done fairly infrequently. E.g. this isn't a case where you have to create a staging environment, a prod environment, a db server, a front-end server, an app server, etc. You create one Streisand instance and occasionally reprovision it. Focusing on supporting the creation of the server seems like the wrong place to spend our scarce resources.

Personally I think the best path forward is:

• Eventually move to Terraform and remove our provider support.
• Export a machine consumable firewall manifest that can be processed by other tools to open/close required ports in provider-specific tooling.
• Improve the "Existing server" support to catch the unsupported providers.

For the first, Terraform support, we get the huge benefit of not having to maintain any cloud provider provisioning code at all. Hashicorp can use their vast resources to support our users that struggle with creating instances/security groups. We focus on Ansible and provisioning existing servers to be Streisand servers.

With the second and third (better existing server support, better non-standard firewall/port support) you can cover all the bases that Terraform doesn't without a ton of additional code/support burden. I think the machine-readable firewall config is the best path forward in this space. For example if you use CPU's VPS Hut and it has a Security-Group-Like mechanism you could write a small script that consumes streisand_firewall.json and uses it to add rules with the CPU VPS Hut API. This can live as a separate tool in a separate repo maintained by separate people. I could conceive of supporting using them automatically as plugins from Streisand one day way down the road.

Anyone want to make the case for why we should keep support for provisioning servers with cloud providers as a Streisand function instead of using a purpose-built tool? I could always be overlooking something :-)

[nickolasclarke]

@cpu is there not an equiv of saltstack's SaltCloud drivers that obfuscate away the integration with popular providers in Ansible? That is essentially the main benefit of moving to Terraform correct? If there is not, then yes, I think that moving to something like saltCloud or terraform to leverage a larger community developing native integrations is the way forward.

Logistically, can we mix and match? Do we completely tear out ansible and replace it with terraform, or use them in a complimentary fashion? I'm not familiar enough. I need to do some reading.

[nopdotcom]

Amazon Lightsail needs ports opened, but that’s it.

I am a little wary of second-system syndrome.

[cpu]

@cpu is there not an equiv of saltstack's SaltCloud drivers that obfuscate away the integration with popular providers in Ansible?

I'm not sure. I only have the way we do things currently to evaluate against and it is definitely not abstracted away very much.

Logistically, can we mix and match? Do we completely tear out ansible and replace it with terraform, or use them in a complimentary fashion? I'm not familiar enough. I need to do some reading.

I'm not sure yet. I haven't put significant time into thinking about what Teraform would require. I suspect mixing and matching is not desirable long term but might be a good way to transition.

Amazon Lightsail needs ports opened, but that’s it. I am a little wary of second-system syndrome.

@nopdotcom Can you elaborate? I don't think the Terraform discussion is specific to Lightsail.

[nopdotcom]

@cpu I was just amplifying that most providers don't need too much care to run Streisand. If I cared enough about Lightsail, I'd probably just write a script to use the AWS CLI to open the ports.