LordAlfredo
commented
4 years ago Installation/configuration behavior was an explicit item discussed with Amazon Linux, Canonical, and Red Hat. The exact decision, as signed off by Amazon leadership and our partners, is that if during install EC2 Instance Connect does not detect that the instance already has a custom AuthorizedKeysCommand then it will attempt to modify the system sshd as part of the installation process. The exact mechanism, however, was disagreed upon by each distro vendor.
- Amazon Linux insisted that it should be modified in the sshd_config directly
- Canonical disagreed with modifying another package's file and instead we agreed to implement a systemd override file
- RHEL+CentOS is still under discussion (see #2, we're still discussing implementation detail with Red Hat)
The detection mechanism is pretty straightforward: during post install scriptlets, check if /etc/ssh/sshd_config already has AuthorizedKeysCommand and AuthorizedKeysCommandUser either not present or as a default #AuthorizedKeysCommand none / #AuthorizedKeysCommandUser nobody
Since you have an AuthorizedKeysCommand configured the post-installation scriptlet should not have installed our override. Can you check if the file /lib/systemd/system/ssh.service.d/ec2-instance-connect.conf exists? If it does then something in the package install/upgrade logic failed.
Deleting this file or uninstalling the ec2-instace-connect package will remove the override.
If you still would like EIC supported alongside your own AuthorizedKeysCommand you will need to add a hook to our scriptlet somewhere in your own - see said override file for how to invoke.
As far as the upgrade itself, did you apply your custom AuthorizedKeysCommand before or after installing/upgrading EIC? Can you provide the apt-get output from when you both upgraded and downgraded?
Along the lines of your suggestion, there is already an explicit end-to-end test that should have captured this - the test specifically configures a custom AuthorizedKeysCommand in sshd_config, then installs EIC, and verifies that EIC did not apply the override. We have an explicit requirement that any proposed code change and package publication have full integration test output for all platforms attached as part of the approval process, so in theory any detection failure should have been caught prior to package release. If we can reproduce the circumstances where you triggered this gap in our detection I would like to refine this test to avoid a future regression.
zscholl
ferricoxide
cregkly
mbainter
mechastorm
kylet21
Issue Summary
When we upgraded the
ec2-instance-connectpackage from 1.1.9 to 1.1.12 on Ubuntu 16.04 and 18.04, ec2-instance-connect overrode ourAuthorizedKeysCommandandAuthorizedKeysCommandUser.For our use case we want to support two sets of
AuthorizedKeysCommandI believe it was this commit which corrected the systemd service file that introduced this issue for us.
Our config
Here is what our
sshd_configlooked like (some values replaced).auth.logWhen attempting our default login path (non-ec2i) we could see that sshd was attempting to use the ec2 instance connect commands to look up authorized keys.
Service details
Solution
We have downgraded and pinned version 1.1.9. This restored our ability to SSH via our directory backed
AuthorizedKeysCommand.Downgrade service details
Suggestions
sshdthrough the service configAuthorizedKeysCommandis respected.AuthorizedKeysCommandon install. Instead print a message that tells users how to configuresshd_configfor themselves.