Lambda -> Git SSH Authentication

UdhavPawar commented 4 years ago

Hello,

Followed Complex example on Node.js w/ ssh steps from README.
Facing issue in authenticating to git over ssh. Verified same creds work locally.
As per guide, getting the private key from AWS SSM and added public key to GIT and /tmp/known_hosts.
Setting environment var for GIT_SSH_COMMAND with UserKnownHostsFile=/tmp/known_hosts and StrictHostKeyChecking=no options.

Python Lambda function:

import os
import subprocess
import boto3
import git

def lambda_handler(event, context):

    # get SSH key
    ssm = boto3.client('ssm')
    parameter = ssm.get_parameter(Name='gitLambda')
    gitLambdaKey = parameter['Parameter']['Value']

    # save SSH key in /tmp and chmod permissions
    with open('/tmp/gitLambdaKey', 'w') as outfile:
        outfile.write(gitLambdaKey)
    os.chmod('/tmp/gitLambdaKey', 0o400) # leading 0 in python2 and 0o in python 3 denies octal

    # clean up /tmp and make dir for repo
    os.system("rm -rf /tmp/* ; mkdir /tmp/git")
    # To fix "Warning: Remote Host Identification Has Changed" error clear known hosts for github.com first then add new
    os.system('ssh-keygen -f "/tmp/known_hosts" -R "github.com"')

    with open('/tmp/known_hosts', 'w') as outfile:
        outfile.write("github.com,192.30.252.*,192.30.253.*,192.30.254.*,192.30.255.*,140.82.113.* ssh-rsa AAA ... cU=")

    os.environ['GIT_SSH_COMMAND'] = "ssh -o UserKnownHostsFile=/tmp/known_hosts -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey"

    try:
        cmd = 'ssh -v git@github.com'
        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True, universal_newlines=True)
    except subprocess.CalledProcessError as exc:
        print("Status FAIL:", exc.returncode, exc.output)
    else:
        print("Output: \n{}\n".format(output))

Error with public key in /tmp/known_hosts:

debug1: Reading configuration data /opt/etc/ssh/ssh_config
debug1: /opt/etc/ssh/ssh_config line 58: Applying options for *
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to github.com [140.82.113.4] port 22.
debug1: Connection established.
debug1: SELinux support disabled
Could not create directory '/home/sbx_user1051/.ssh'.
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version babeld-5a455904
debug1: no match: babeld-5a455904
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8
debug1: read_passphrase: can't open /dev/tty: No such file or directory
Host key verification failed.

Any guidance is appreciated. Thank You.

mhart commented 4 years ago

It looks like git is trying to read a passphrase – is your key signed with a passphrase perhaps?

UdhavPawar commented 4 years ago

It looks like git is trying to read a passphrase – is your key signed with a passphrase perhaps?

@mhart thanks for looking into this. Initially thought same. Verified by creating a key with no passphrase (pressing enter during keygen). Also, verified locally

udhavpawar$ ssh -T -i gitLambda git@github.com
Hi UdhavPawar! You've successfully authenticated, but GitHub does not provide shell access.

mhart commented 4 years ago

Actually, the read_passphrase: can't open /dev/tty: No such file or directory line might be a red herring – it could be completely unrelated.

Are you sure that the key is correct? Can you reproduce this locally (by writing the key from SSM to a file in exactly the same way?)

mhart commented 4 years ago

Hang on – I've just realized – the command you're executing with subprocess is ssh -v git@github.com 🤦

That's not git! ssh doesn't know anything about the GIT_SSH_COMMAND environment variable – only git uses this.

If you want to test ssh, then you should specify all those other options on the command line too:

cmd = 'ssh -v -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey -T git@github.com'

UdhavPawar commented 4 years ago

@mhart Was following instructions process.env.GIT_SSH_COMMAND = 'ssh -o UserKnownHostsFile=/tmp/known_hosts -i /tmp/id_rsa' mentioned on lambci's README.md.

Updated lambda function's command with git ssh command options


import os
import subprocess
import boto3
import git
import re

def lambda_handler(event, context):

# get SSH key
ssm = boto3.client('ssm')
parameter = ssm.get_parameter(Name='gitLambda')
private_key = parameter['Parameter']['Value']

# save SSH key in /tmp and chmod permissions
with open('/tmp/gitLambdaKey', 'w') as outfile:
    outfile.write(private_key)
os.chmod('/tmp/gitLambdaKey', 0o400) # leading 0 in python2 and 0o in python 3 defines octal

# clean up /tmp and make dir for repo
os.system("rm -rf /tmp/* ; mkdir /tmp/git")
# To fix - Warning: Remote Host Identification Has Changed error clear known hosts for github.com first then add new
os.system('ssh-keygen -f "/tmp/known_hosts" -R "github.com"')

with open('/tmp/known_hosts', 'w') as outfile:
    outfile.write("github.com,192.30.252.*,192.30.253.*,192.30.254.*,192.30.255.*,140.82.113.* ssh-rsa AAA...uScU= udhavpawar@udhav.pawar")

os.environ['GIT_SSH_COMMAND'] = "ssh -o UserKnownHostsFile=/tmp/known_hosts -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey"

try:
    cmd = 'ssh -v -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey -T git@github.com'
    output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True, universal_newlines=True)
except subprocess.CalledProcessError as exc:
    print("Status FAIL:", exc.returncode, exc.output)
else:
    print("Output: \n{}\n".format(output))

- verified locally, authentication works using same SSH key

udhav:.ssh udhavpawar$ ssh -o StrictHostKeyChecking=no -i ./gitLambdaKey -T git@github.com Hi UdhavPawar! You've successfully authenticated, but GitHub does not provide shell access.

- But lambda still throws same error

START RequestId: 79589a04-830d-4a48-af70-ded48f90e6bb Version: $LATEST do_known_hosts: hostkeys_foreach failed: No such file or directory Status FAIL: 255 Warning: Identity file /tmp/gitLambdaKey not accessible: No such file or directory. OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 debug1: Reading configuration data /opt/etc/ssh/ssh_config debug1: /opt/etc/ssh/ssh_config line 58: Applying options for * debug1: Connecting to github.com [140.82.112.3] port 22. debug1: Connection established. debug1: SELinux support disabled Could not create directory '/home/sbx_user1051/.ssh'. debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_rsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/sbx_user1051/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4 debug1: Remote protocol version 2.0, remote software version babeld-b447314b debug1: no match: babeld-b447314b debug1: Authenticating to github.com:22 as 'git' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: rsa-sha2-512 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ssh-rsa SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 Failed to add the host to the list of known hosts (/home/sbx_user1051/.ssh/known_hosts). debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/sbx_user1051/.ssh/id_rsa debug1: Trying private key: /home/sbx_user1051/.ssh/id_dsa debug1: Trying private key: /home/sbx_user1051/.ssh/id_ecdsa debug1: Trying private key: /home/sbx_user1051/.ssh/id_ed25519 debug1: No more authentication methods to try. Permission denied (publickey).

END RequestId: 79589a04-830d-4a48-af70-ded48f90e6bb REPORT RequestId: 79589a04-830d-4a48-af70-ded48f90e6bb Duration: 1591.44 ms Billed Duration: 1600 ms Memory Size: 128 MB Max Memory Used: 79 MB Init Duration: 249.79 ms



lmk if any additional info is needed. ty.

mhart commented 4 years ago

The error says:

Status FAIL: 255 Warning: Identity file /tmp/gitLambdaKey not accessible: No such file or directory.

Because you delete the file after you create it!

os.system("rm -rf /tmp/* ; mkdir /tmp/git")

Also – you might want to regenerate that private key that you posted initially, if you haven't already

UdhavPawar commented 4 years ago

Thanks for pointing that out @mhart . Giving it a last try.

Updated Lambda function:

import os
import subprocess
import boto3
import git
import re

def lambda_handler(event, context):
    # clean up /tmp
    os.system("rm -rf /tmp/*") 

    # get SSH key
    ssm = boto3.client('ssm')
    parameter = ssm.get_parameter(Name='gitLambda')
    private_key = parameter['Parameter']['Value']

    # save SSH key in /tmp and chmod permissions
    with open('/tmp/gitLambdaKey', 'w') as outfile:
        outfile.write(private_key)
    os.chmod('/tmp/gitLambdaKey', 0o400) # leading 0 in python2 and 0o in python 3 defines octal

    # make dir for repo
    os.system("mkdir /tmp/git")
    # To fix - Warning: Remote Host Identification Has Changed error clear known hosts for github.com first then add new
    os.system('ssh-keygen -f "/tmp/known_hosts" -R "github.com"')

    with open('/tmp/known_hosts', 'w') as outfile:
        outfile.write("github.com,192.30.252.*,192.30.253.*,192.30.254.*,192.30.255.*,140.82.113.* ssh-rsa private-key")

    os.environ['GIT_SSH_COMMAND'] = "ssh -o UserKnownHostsFile=/tmp/known_hosts -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey"

    try:
        cmd = 'ssh -v -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey -T git@github.com'
        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True, universal_newlines=True)
    except subprocess.CalledProcessError as exc:
        print("Status FAIL:", exc.returncode, exc.output)
    else:
        print("Output: \n{}\n".format(output))

Error still says:

START RequestId: 1e534932-20f4-4995-b7b4-8b45ba1c031c Version: $LATEST
do_known_hosts: hostkeys_foreach failed: No such file or directory
Status FAIL: 255 OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /opt/etc/ssh/ssh_config
debug1: /opt/etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to github.com [140.82.112.3] port 22.
debug1: Connection established.
debug1: SELinux support disabled
Could not create directory '/home/sbx_user1051/.ssh'.
debug1: key_load_public: No such file or directory
debug1: identity file /tmp/gitLambdaKey type -1
debug1: key_load_public: No such file or directory
debug1: identity file /tmp/gitLambdaKey-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version babeld-b447314b
debug1: no match: babeld-b447314b
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8
Failed to add the host to the list of known hosts (/home/sbx_user1051/.ssh/known_hosts).
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/gitLambdaKey
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug1: No more authentication methods to try.
Permission denied (publickey).

mhart commented 4 years ago

Looks like the key isn't valid

mhart commented 4 years ago

You could try running in a more verbose mode (-vv or -vvv) to see if there's more details

UdhavPawar commented 4 years ago

@mhart after creating a fresh key, tried diff methods:

using manual key with lambda funciton.
fetching it from AWS SSM, storing it in /tmp dir.

But none of the methods have worked so far. lmk if I should try any other method or any other reference readme. TY for your guidance and time.

UdhavPawar commented 4 years ago

Waited for reply, but I guess team is busy. Will figure out work-around. TY @mhart for your guidance and time.

adriancuero commented 1 year ago

Hi @UdhavPawar ! Did you figure out any work-around? Thanks in advance.

UdhavPawar commented 1 year ago

Hi @UdhavPawar ! Did you figure out any work-around? Thanks in advance.

@adriancuero Tried but it didn't work. My usecase was running some automated git actions in lambda, but now that's supported by default with git workflow actions. Check if that fits your bill

lambci / git-lambda-layer

Lambda -> Git SSH Authentication #26