Upload all deployment keys at once

[mbrgm]

Motivation

When there are multiple deployment.keys, a separate scp upload is done for every single key file. This takes a lot of time and could probably be faster.

Idea

Maybe the keys could be uploaded all at once -- either using recursive scp -r on the whole keys directory or by some other means.

[domenkozar]

We could just list all keys and scp would upload them in one go

[mbrgm]

With 8f4a67ca591f9d127344bca3ecd752d3d97a716d in place, we would now have to use one scp per destDir, as scp accepts multiple input files, but only one target path.

@domenkozar What do you think about the following idea (which is heavily inspired by how saltstack runs remote commands)?

1. Create a .sh script locally, which, when run on the target creates the key files in their appropriate destDirs. Content can be base64 encoded to avoid escaping issues or allow binary content.
2. scp that script to the target.
3. Run transferred script on the target.
4. Delete script on the target.

[moretea]

NixOps uses an SSH master socket. It might be sufficient to do something like this (pseudocode)

for secret in secrets:
    handle = master_ssh.exec_shell("cat > /secrets/file")
    handle.stdin.write(secret)

[domenkozar]

With 7 keys and being in asia this is quite painful :) Will give it a try.

[domenkozar]

Don't think we should further reinvent paramiko, need to resurrect https://github.com/NixOS/nixops/pull/124

[grahamc]

We were spawning way too many SSH connections at once, and reverted in #1269.

[domenkozar]

Not 100% the issue are connections, I could imagine a session is a process so he was seeing 100 session processes with 10 connections. Needs investigation but should be easy to reproduce.