Extract the SSH/WinRM communicators and make them a library in their own right

[MihaiBojin]

Current Terraform Version

2019/06/20 12:28:34 [INFO] Terraform version: 0.11.11  
2019/06/20 12:28:34 [INFO] Go runtime version: go1.11
2019/06/20 12:28:34 [INFO] CLI args: []string{"/snap/terraform/216/bin/terraform", "-v"}
2019/06/20 12:28:35 [INFO] CLI command args: []string{"version", "-v"}
Terraform v0.11.11

Use-cases

Terraform's SSH communicator is a great tool with several useful features (connection through a bastion, ssh/scp, ssh-agent support, etc.) When looking for an ssh wrapper on top of go's native libraries, I found that the ssh/communicator was the one closest to what I needed to do.

However, importing it directly from Terraform 'feels' a bit strange and moreover contributing other features that do not necessarily match what a TF provisioner should do, also feels off.

I am proposing that you extract both communicators into a separate library so that they can be iterated on and improved upon.

One example of a missing feature is to download a file from a remote VM. Another missing feature is to establish local port forwarding.

The advantage and potential use of the communicator as a standalone library would, for example, be to create a new generation of providers which mimick what a provisioner does, but do so using the benefits of Terraform's resource management model, allowing managed state updates without having to re-provision the underlying infrastructure.

For example, take the following actions:

• provision a VM
• provision a package (resource) (e.g. apt install -y some-package)
• update a package (resource) (e.g. apt install -y some-package=some-version)
• delete a package (resource) (e.g. apt remove some-package)
• configure the package
• etc.

Currently, the above is achieved by provisioning a VM using, for example, the AWS provider and then adding a long list of bash commands to provision the necessary packages. If any of these needs to be updated, reconfigured, etc., the VM would have to be destroyed and recreated - which is an expensive operation.

State changes could be more efficiently handled by a provider executing the necessary steps via SSH/WinRM.

A first step to enable authors to write such providers would be to have a great standalone communicator.

Thank you!

Attempted Solutions

Example: downloading a file from a VM - can be achieved with the ssh communicator by issuing a cat /path/to/file and capturing the output. For binary files, the base64 utility would have to be available on the target VM. This is cumbersome and inefficient.

Example 2: check if a port is open on the VM - can be achieved with the ssh communicator via nc -zv 127.0.0.1 221 (but required netcat); some other workarounds exist (such as cat < /dev/null > /dev/tcp/127.0.0.1/PORT - but they are awkward).

A much better solution would be to build these features into the SSH/WinRM communicators.

Of course, a very obvious alternative would be to add these features directly into terraform's communicator package - in which case this issue can be solved.

Proposal

See above.

References

N/A

[apparentlymart]

Hi @MihaiBojin! Thanks for requesting this.

We have no plans right now to split out that functionality into a separate codebase. As you expected, we don't consider it a public interface and aren't ready to take on the burden of maintaining API compatibility between releases for the benefit of external callers. In particular, it's likely that some details about how the provisioners work will evolve in a future release as we work towards splitting them out into separate plugins, and we don't know yet how that will impact the different components involved in provisioners.

However, that code is covered by Terraform's MPL v2.0 license, which permits you to take your own copy of the code and adapt it into a library yourself, if you'd like. That would allow you to get the library functionality you need without the risk of Terraform-imposed breaking changes in the future. Of course, it would also make you the primary maintainer of that library, which has both advantages and disadvantages.

[MihaiBojin]

Thank you, @apparentlymart ! Knowing your plans helps with making the decision of taking it out into a separate library.

Can you point me to an example of a library which came out of TF? Or a guide on how to achieve this without infringing the license? (I think it would be easier to see an example) Alternatively, can I add you to the initial PR for a review?

Thank you for all your help and guidance!

[apparentlymart]

Hi @MihaiBojin!

The license does include some "responsibilities" under which the code can be distributed, but they are generally consistent with other open source licenses. If you plan to release your own library as open source too then they shouldn't require any unusual steps. If you make your library open source and use the same license then the main thing would be a note in your documentation that the code was derived from Terraform, as attribution.

Although it's been a long time since the splits happened, the libraries HIL and HCL both originated from functionality inside Terraform. They are both maintained by HashiCorp itself, but you'll see that they are both still under the MPL v2.

[MihaiBojin]

Thank you!

[shinenelson]

Can't this issue be closed? It has been open for 2 years. Previous comments indicate a wontfix status.