"error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ...snip... failed, status 22" error

[yoshifumi-kinoshita]

Because of the combination of shell option set -e and curl -f, curl returns 22 when HTTP response is 4xx or 5xx, and eic_curl_authorized_keys stops immediately, and makes the following error in /var/log/secure.

error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ...(snip)...  failed, status 22

Repro steps

1. Create Amazon Linux 2 instance.
2. Install ec2-instance-connect-1.1-12.amzn2.noarch.
3. Logout
4. Login via ssh
5. grep error /var/log/secure

   May  8 19:03:03 ip-172-31-46-138 sshd[2565]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ec2-user SHA256:g7uVwGWoozM/n3GrFbjJ8kYyIF7A7UawA3nbRQDxbxk failed, status 22
   May  8 19:12:32 ip-172-31-46-138 sshd[2552]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ec2-user SHA256:g7uVwGWoozM/n3GrFbjJ8kYyIF7A7UawA3nbRQDxbxk failed, status 22
   May  8 19:13:30 ip-172-31-46-138 sshd[2629]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ec2-user SHA256:g7uVwGWoozM/n3GrFbjJ8kYyIF7A7UawA3nbRQDxbxk failed, status 22

[trevorlatson]

This is still an issue...

[arun6445]

I keep getting the same error as well

[AravindTReddy]

Just faced with the same issue. Any solution ? Thanks

[marcfreiheit]

I faced the same issue, as well. For me, wrong ownership of the ~/.ssh directory caused the issue. Permissions were set properly to 700 for ~/.ssh and 600 to ~/.ssh/authorized_keys, but the owner was root instead of the user logging in.

I don't know about the details of that fail code, but I suspect that it might be related to insufficient permissions to read any kind of configuration or keys.

[dallasmarlow]

@marcfreiheit I also observed this issue just now, but the owner of the /home/ec2-user/.ssh directory and authorized_keys file owner was correctly set to ec2-user.ec2-user

[dhruv-malik-ptc]

I too am facing this issue with new AL2 images

[corby]

Bump this. It's messing with our intrusion detection software because it shows as an SSHD error (when it's not).

I was able to fix this by uninstalling the package, but I'd rather not have to do this for every instance we run

[deogracia]

still an issue

• env

LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: Amazon
Description:    Amazon Linux release 2 (Karoo)
Release:    2
Codename:   Karoo

steps done

• create a fresh amazon linux 2 instance

• run upgrade

• create a user

• create an ed25519 key pair via ssh-keygen with this new user

• try to connect from my pc with the newly created private key fails with AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys <USER> SHA256:<HASH> failed, status 22

• Removing ec2-instance-connect, still can't connect and the only message logged is Closed due to user request. [preauth]

[deogracia]

still an issue

* env

LSB Version:  :core-4.1-amd64:core-4.1-noarch
Distributor ID:   Amazon
Description:  Amazon Linux release 2 (Karoo)
Release:  2
Codename: Karoo

steps done

* create a fresh amazon linux 2 instance

* run upgrade

* create a user

* create an ed25519 key pair via ssh-keygen with this new user

* try to connect from my pc with the newly created private key fails with ` AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys <USER> SHA256:<HASH> failed, status 22`

* Removing `ec2-instance-connect`, still can't connect and the only message logged is `Closed due to user request. [preauth]`

Sorry: mine was a configuration error (I forgot to add the pub key in authhorized_keys). So not an EIC issue.

[loebpaul]

I'm still having the same issue. Every time I login via ssh I get this error in my log. Anyone have a solution?

sshd: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ec2-user SHA256 failed, status 22

[marcleblanc2]

I'm getting this too, and I'm not using EC2 Instance Connect.

[dhruv-malik-ptc]

I'm getting this too, and I'm not using EC2 Instance Connect.

Mine was fixed, trying to recall how Are you trying to connect AFTER hardening? Check your sshd_config file, for the below parameter, if you are using a local user with a password, then this option needs to be 'yes', and it requires a restart of sshd service.

PasswordAuthentication yes

Which step are you facing error in? I have been able to harden the image and get it to work.

[mjeffe]

I have the same issue on every new Amazon Linux 2 box I bring up. I don't use instance connect, so sudo yum remove ec2-instance-connect solves it for me. But I'd rather leave it installed and fix the problem since someday I may need it.

[exNewbie]

I have the same issue on every new Amazon Linux 2 box I bring up. I don't use instance connect, so sudo yum remove ec2-instance-connect solves it for me. But I'd rather leave it installed and fix the problem since someday I may need it.

Removing ec2-instance-connect doesn't resolve the problem for me.

[exNewbie]

I got the similar issue and found more details about the problem.

On /var/log/message, I found

error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Apr 10 00:52:21 app sshd[3340]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Apr 10 00:52:21 app sshd[3340]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Apr 10 00:52:21 app sshd[3340]: sshd: no hostkeys available -- exiting.

This blocks SSH access to the server. The cause is because of a bug/feature on cloud-init which is reported at https://bugzilla.redhat.com/show_bug.cgi?id=1888761 https://bugs.centos.org/view.php?id=18046

Long story short, cloud-init deletes SSH host keys in order to prevent repeative host keys when a server image is cloned.

To fix this, I overwrite value of ssh_deletekeys on /etc/cloud/cloud.cfg and set it to false

ssh_deletekeys:   false

Nonetheless, there are other lazy workarounds

• reboot the server
• have a script on rc to check whether SSH host keys exist. If not, generate them

[cam8001]

For what its worth, this error happens for me on AL2 on ARM when trying to connect from my local laptop via ssm/ssh proxy command.

When I launched an instance on the same version of AL2 but on x86-64, it worked fine.

I could still connect to the ARM instance via the ssm interface in the web console, just not via ssh.

[ohitspaul]

Is this still an issue? What AL2 images are causing this issue, and what regions are you launching in? It seems like many of these issues are due to misconfigurations (missing host keys, not having the public key in authorized_keys file).

[dhruv-malik-ptc]

Anyone facing this issue should make sure you do not remove cloud-init package during ami baking process.

[trogau]

Is this still an issue? What AL2 images are causing this issue, and what regions are you launching in? It seems like many of these issues are due to misconfigurations (missing host keys, not having the public key in authorized_keys file).

I'm still seeing this error on a relatively fresh (~3 week old) Lightsail VPS running AL2 (ap-southeast-2a). I've created two new users on the server and I get error lines referencing both users, although not for the built-in ec2-user account. Both users have public keys in their authorized_keys file. Users are chroot jailed (as a group) but that's the only other change I've made to ssh setup.

[ohitspaul]

@trogau in what cases do you see the error lines appear? (e.g. attempting to SSH using the public keys in the authorized_keys file, attempting to use EC2 Instance Connect to push public keys for the users and attempting to SSH with those keys, or some other scenario)

[trogau]

Apologies, should have included some logging info. This is what I see in /var/log/secure:

Jul  6 20:32:24 ip-172-26-x-xx sshd[26406]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 20:32:24 ip-172-26-x-xx sshd[26406]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 20:32:24 ip-172-26-x-xx sshd[26406]: Accepted publickey for user2 from {{ip}} port 49899 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 20:32:24 ip-172-26-x-xx sshd[26406]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 20:32:24 ip-172-26-x-xx sshd[26406]: pam_unix(sshd:session): session closed for user user2
Jul  6 20:47:21 ip-172-26-x-xx sshd[26502]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 20:47:21 ip-172-26-x-xx sshd[26502]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 20:47:21 ip-172-26-x-xx sshd[26502]: Accepted publickey for user2 from {{ip}} port 50137 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 20:47:21 ip-172-26-x-xx sshd[26502]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 20:47:21 ip-172-26-x-xx sshd[26502]: pam_unix(sshd:session): session closed for user user2
Jul  6 21:02:26 ip-172-26-x-xx sshd[26583]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:02:26 ip-172-26-x-xx sshd[26583]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:02:26 ip-172-26-x-xx sshd[26583]: Accepted publickey for user2 from {{ip}} port 50371 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 21:02:26 ip-172-26-x-xx sshd[26583]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 21:02:26 ip-172-26-x-xx sshd[26583]: pam_unix(sshd:session): session closed for user user2
Jul  6 21:17:18 ip-172-26-x-xx sshd[26688]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:17:18 ip-172-26-x-xx sshd[26688]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:17:18 ip-172-26-x-xx sshd[26688]: Accepted publickey for user2 from {{ip}} port 50636 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 21:17:18 ip-172-26-x-xx sshd[26688]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 21:17:18 ip-172-26-x-xx sshd[26688]: pam_unix(sshd:session): session closed for user user2
Jul  6 21:32:21 ip-172-26-x-xx sshd[26792]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:32:21 ip-172-26-x-xx sshd[26792]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:32:21 ip-172-26-x-xx sshd[26792]: Accepted publickey for user2 from {{ip}} port 50860 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 21:32:21 ip-172-26-x-xx sshd[26792]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 21:32:21 ip-172-26-x-xx sshd[26792]: pam_unix(sshd:session): session closed for user user2
Jul  6 21:47:20 ip-172-26-x-xx sshd[26852]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:47:20 ip-172-26-x-xx sshd[26852]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 21:47:20 ip-172-26-x-xx sshd[26852]: Accepted publickey for user2 from {{ip}} port 51112 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 21:47:20 ip-172-26-x-xx sshd[26852]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 21:47:21 ip-172-26-x-xx sshd[26852]: pam_unix(sshd:session): session closed for user user2
Jul  6 22:02:27 ip-172-26-x-xx sshd[26970]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 22:02:27 ip-172-26-x-xx sshd[26970]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 22:02:27 ip-172-26-x-xx sshd[26970]: Accepted publickey for user2 from {{ip}} port 51353 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 22:02:27 ip-172-26-x-xx sshd[26970]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 22:02:27 ip-172-26-x-xx sshd[26970]: pam_unix(sshd:session): session closed for user user2
Jul  6 22:05:25 ip-172-26-x-xx sshd[27009]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 22:05:25 ip-172-26-x-xx sshd[27009]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 22:05:25 ip-172-26-x-xx sshd[27009]: Accepted publickey for user2 from {{ip}} port 51390 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 22:05:25 ip-172-26-x-xx sshd[27009]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 22:05:25 ip-172-26-x-xx sshd[27009]: pam_unix(sshd:session): session closed for user user2
Jul  6 22:06:11 ip-172-26-x-xx sshd[27048]: Accepted publickey for ec2-user from 120.88.121.179 port 58361 ssh2: RSA SHA256:Gxc8C5pq8XLh4Sk7rOvPvp4tDsmmhIkavVXAY90E1E4
Jul  6 22:06:11 ip-172-26-x-xx sshd[27048]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
Jul  6 22:06:13 ip-172-26-x-xx sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/su -
Jul  6 22:06:13 ip-172-26-x-xx sudo: pam_unix(sudo:session): session opened for user root by ec2-user(uid=0)
Jul  6 22:06:13 ip-172-26-x-xx su: pam_unix(su-l:session): session opened for user root by ec2-user(uid=0)
Jul  6 22:17:23 ip-172-26-x-xx sshd[27130]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 22:17:23 ip-172-26-x-xx sshd[27130]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user2 SHA256:{{keyFingerprint}} failed, status 22
Jul  6 22:17:23 ip-172-26-x-xx sshd[27130]: Accepted publickey for user2 from {{ip}} port 51619 ssh2: RSA SHA256:{{keyFingerprint}}
Jul  6 22:17:23 ip-172-26-x-xx sshd[27130]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jul  6 22:17:24 ip-172-26-x-xx sshd[27130]: pam_unix(sshd:session): session closed for user user2

The users are connecting via SFTP, not ssh'ing into the VPS, if that makes a difference. I see lines like this for both the two users I added to the box manually & set up keys for, but as noted above I do /not/ see any for the built-in ec2-user account. Happy to diagnose further (I am AEST timezone, so apologies for the slow reply :)

[SpComb]

This error is logged for any user that does not have active managed-ssh-keys available at the time of login:

$ sudo -u ec2-instance-connect sh -x /opt/aws/bin/eic_curl_authorized_keys root || echo $?
...
++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token: ...' -o /dev/null -I -w '%{http_code}' http://169.254.169.254/latest/meta-data/managed-ssh-keys/active-keys/root/
+ keys_status=404
22

The last status update on this from AWS seems to be a comment from @CptTZ in the related community PR, one year ago: https://github.com/aws/aws-ec2-instance-connect-config/pull/21#issuecomment-651528795

Closing as we are currently going through a internal process for this change

Has this been fixed internally yet?

[LeadManPL]

I confirm this is happening only for users I create manually from OS level, in chrooted sftp only group, with Pub/Sec key pairs generated. Does anyone know how to notify aws software that OS users has PK genreated?

[LeadManPL]

Solution: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-user-account-cloud-init-user-data/

In my case in "View/Change User Data." I had to add:

users:
  - name: sftpuser
    groups: [ sftponly ]
    sudo: [ "ALL=(ALL) NOPASSWD:ALL" ]
    shell: /bin/false
    ssh-authorized-keys: 
    - ssh-rsa AAAAB3NzaC1yc2E[ here_goes_rest_of_your_PubKey ] sftp_user

Where: sftpuser - user I created manually on OS level sftponly - group I created manually on OS level

[rofc]

Recently I faced this error and it was related to ssh-agent, which was trying to send different keys (take a look on ssh-add -l output and compare hashes to the ones on the /var/log/secure in your instance) instead the specified one in command line (using "-i" option). You can add -o "IdentitiesOnly yes" to your command line (or ssh config file).

[biohazd]

im also seeing this use, is there a fix yet ?

[sushidub]

I dealt with this exact error all day today. It turned out that by changing ownership of not only the .ssh/ directory but the authorized_keys file as well - from root:root to ec2-user:ec2-user finally fixed the issue. There's a good chance I may have made a number of other potential fixes or corrected configuration settings during troubleshooting tho.

IMHO Permission Denied, status 22messaging might be semantically correct in its report but unfortunately is lacking in context - which seems like a fairly trivial effort to provide it.

[darox]

I have this issue despite the fact that the user is ec2-user and permissions 700 and 600.

[LeadManPL]

@darox : see my workaround above

[TrentonAdams]

I am having the same issue, yet my co-worker is not, on the same instance.

I noted that it never got me into the ec2-user, and dropped me into an unconfigured shell.

sh-4.2$ whoami
ssm-user

I can...

sh-4.2$ sudo su - ec2-user
Last login: Thu Dec  9 04:56:37 UTC 2021 on pts/1
[ec2-user@ip-10-0-0-118 ~]$

The above happens when using aws ssm start-session --target i-07fb6163f7428c2c0 or when doing it from the management console.

This works perfectly, but requires ssh port access...

mssh -i i-07fb6163f7428c2c0

Another thing to note is that I used aws-cdk's Bastion host construct to create it.

        const host = new BastionHostLinux(this, 'BastionHost', {
            vpc: this.vpc,
            subnetSelection: {
                subnetType: SubnetType.PUBLIC
            },
            blockDevices: [
                {
                    deviceName: '/dev/xvda',
                    volume: BlockDeviceVolume.ebs(10, {
                        encrypted: true,
                    }),
                }],
        });

[ugur1yildiz]

I am using lightsail instance and ssh via ec2-user and I get the same error but login successfully. I tried all the recommendations above but not LeadManPL's solution since Lightsail instances don't have "View/Change User Data" action.

I think it is safe to remove ec2-instance-connect which lightsail don't support

ec2-instance-connect installation Arch : noarch Version : 1.1 Release : 15.amzn2

[Lunartist]

Bump. So annoying.

[peterhodes]

I found this problem too - simple fix is to change the Key Pair Type from "ED25519" to "RSA". I realise this is not a great fix as if you're using ED25519 then you likely have good reasons to do so - BUT - knowing this is the fix might give you an angle on figuring out the actual cause. Unfortunately I don't have time to look at that right now as I'm working on delivering something - but hopefully that might help you figure it out.

Good luck. Peter Rhodes.

[plumbuma]

Also ran into this issue where I could not log in while the permissions of the remote .ssh folder and authorized_keys file were 'to high'. I tried many more variations, but for example purposes: chmod -R 777 .ssh FAILS chmod -R 766 .ssh FAILS chmod -R 755 .ssh Works

Found the following in /var/log/auth.log at the time of the error: _AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorizedkeys SHA256: failed, status 22

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal 

[mike-kiwi]

I think plumbuma is on the right track, but do not use the -R or you could might change permissions on the private key(s) (which must be mode 600) or make things executable.

My problem was fixed with this (was mode 775):

chmod 700 ~/.ssh

[cidrbl0ck]

This still seems to be a problem.. with no response from anyone official. Happening today on an Ubu 22.04 LTS ami, local user added via script. .ssh = 700 authorized_keys = 600, correct owner:group too.

[Seshiria]

I'm having a similar issue where the web ssh in lightsail is not connecting. The system is ubuntu2204,I found that the newly installed machine is not having connection problems. I compared the config files and found that: The following is missing from /etc/sshd/sshd_config TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub

Restarted sshd after adding it, problem solved.

It looks like /etc/sshd/sshd_config was overwritten during an apt upgrade somewhere.

[ekkeguembel]

Obviously a very generic error message. I also had the "works for all but one user" situation. Tried a lot of the routes above - in the end it turned out that simply the public key in authorized_keys did not match the private key which was used. So simple ;)

[mparksGP]

I have a 2204 machine that was working fine and suddenly started denying connections with this error sometime in the last seven days, already checked permissions and such.