Secure Sockets - Githubissues

Draft to support high-performance socket connections in Livy ( https://github.com/rstudio/sparklyr/issues/1579) and also other job-spawning technologies.

Overview: Livy has been really popular in the sparklyr community, mostly out of necessity since it enables connectivity for tightly managed clusters or clusters that ran out of capacity to install additional boundary machines were RStudio could be installed, etc. While Livy has proved to be a good solution for connectivity, it's performance is less than ideal since it's primary purpose is to provide a remote REPL.

Solution: One solution would be to request system administrators to make a few additional ports available for sparklyr to upgrade the HTTP connection into a high-performance binary connection sparklyr already supports. However, some additional work is required to provide proper secure access and to the potentially internet visible ports.

Work Items:

[ ] Support to generate private key in Scala or R, consider KeyPairGenerator or PKI.
[ ] Support to exchange private/public keys through Livy. If Livy is compromised, one could attack directly the Livy connection anyways.
[ ] Support to encrypt/decrypt data in Scala, consider RSA Cypher.
[ ] Support to encrypt/decrypt data in R, consider PKI.
[ ] Support to start sparklyr backend in Livy, this is already supported when using spark_apply() with Livy but not currently initialized in the driver node.
[ ] Support to upgrade HTTP Livy connection, when possible.

Notes: This port upgrade would be supported in Amazon AWS since a security group can be defined; however, other platforms like Azure HDInsight does not seem to support port configuration.

sparklyr / sparklyr

Secure Sockets #1595