ubuntu-latest ssh issue in packer proxy adapter for ansible

dendle commented 1 year ago

Description

I have been using ubuntu-latest in my azure devops pipeline that runs packer. I noticed that it had stopped working yesterday (2022/12/13). The error occurred when packer was connecting to an AWS instance of ubuntu.
The SSH session was created successfully, but when packer tries to set up a proxy adapter for ansible to use, it fails to connect with the following error: no matching host key type found. Their offer: ssh-rsa This happens if the target ubuntu instance is 20.04 or 22.10.

This problem goes away if I specify

pool:
  vmImage: Ubuntu-20.04

Platforms affected

[X] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[X] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

20221212.1

Is it regression?

yes, last working image version 20221125.1

Expected behavior

I expected the azure devops hosted runner to be able to execute the packer pipeline without ssh errors.

Actual behavior

An ssh error is encountered when attempting to connect to Ubuntu 20.04 or Ubuntu 22.10. When the packer pipeline runs, it creates an AWS EC2 instance of ubuntu, and attempts to connect to it using SSH. Whilst the initial SSH connection works, when using packer with ansible, there is an SSH "proxy adapter" that is used to connect to the instance.

When ansible attempts to connect via this proxy adapter, the following ssh error occurs: Short version: Unable to negotiate with 127.0.0.1 port 34915: no matching host key type found. Their offer: ssh-rsa Long version:

Failed to connect to the host via ssh: OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/vsts/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/vsts/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/home/vsts/.ansible/cp/fd8cae07e5\" does not exist\r\ndebug3: ssh_connect_direct: entering\r\ndebug1: Connecting to 127.0.0.1 [127.0.0.1] port 34915.\r\ndebug3: set_sock_tos: set socket 3 IP_TOS 0x10\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 10000 ms remain after connect\r\ndebug1: identity file /tmp/ansible-key3025256565 type -1\r\ndebug1: identity file /tmp/ansible-key3025256565-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3\r\ndebug1: Remote protocol version 2.0, remote software version Go\r\ndebug1: compat_banner: no match: Go\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 127.0.0.1:34915 as 'vsts'\r\ndebug3: put_host_port: [127.0.0.1]:34915\r\ndebug1: load_hostkeys: fopen /home/vsts/.ssh/known_hosts: No such file or directory\r\ndebug1: load_hostkeys: fopen /home/vsts/.ssh/known_hosts2: No such file or directory\r\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\r\ndebug3: order_hostkeyalgs: no algorithms matched; accept original\r\ndebug3: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c\r\ndebug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256\r\ndebug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r\ndebug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r\ndebug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: compression ctos: zlib@openssh.com,zlib,none\r\ndebug2: compression stoc: zlib@openssh.com,zlib,none\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1\r\ndebug2: host key algorithms: ssh-rsa\r\ndebug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr\r\ndebug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr\r\ndebug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96\r\ndebug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96\r\ndebug2: compression ctos: none\r\ndebug2: compression stoc: none\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug1: kex: algorithm: curve25519-sha256\r\ndebug1: kex: host key algorithm: (no match)\r\nUnable to negotiate with 127.0.0.1 port 34915: no matching host key type found. Their offer: ssh-rsa"

Repro steps

Create a pipeline that uses packer and ubuntu-latest:
```
trigger:
```
main

pool: vmImage: ubuntu-latest

steps:

task: PackerTool@0 inputs: version: '1.8.5'
task: AWSShellScript@1 displayName: Packer Build inputs: awsCredentials: 'Build Account (Devops)' regionName: 'eu-west-1' scriptType: 'inline' inlineScript: | ansible --version packer init linux_azure_devops_agent.pkr.hcl packer build -on-error=cleanup -var "image_postfix="$(Build.BuildId) linux_agent.pkr.hcl disableAutoCwd: true'

linux_agent.pkr.hcl:

packer {
  required_plugins {
    amazon = {
      version = ">= 0.0.2"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

variable "image_postfix" {
}

source "amazon-ebs" "myinstance" {
  ami_name      = "vm-${var.image_postfix}"
  instance_type = "t2.micro"
  region        = "eu-west-1"
  vpc_id         = "vpc-id"
  subnet_filter {
    filters = {
          "tag:aws-cdk:subnet-type": "Public"
    }
    most_free = true
    random = false
  }
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/*ubuntu-focal-20.04-amd64-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["amazon"]
  }
  ssh_username = "ubuntu"
}

build {
  name = "build-agent-build"
  sources = [
    "source.amazon-ebs.myinstance"
  ]

    provisioner "ansible" {
      playbook_file = "./playbook.yml"
    }
}

playbook.yml:

---
- name: Basic Playbook
  hosts: all

  tasks:
  - name: Upgrade all apt packages
    apt:
      force_apt_get: yes
      upgrade: dist
    become: yes

Execute the pipeline
Observe SSH error before playbook can run

igorboskovic3 commented 1 year ago

Hi @dendle we will take a look, thanks

mikhailkoliada commented 1 year ago

Hello! It is localhost referenced in your logs, it means nothing we can help with, you should amend the ~/.ssh/config file to accept the specific encryption algorithm as say, described here: https://unix.stackexchange.com/questions/402746/ssh-unable-to-negotiate-no-matching-key-exchange-method-found

dendle commented 1 year ago

Hi! As I've mentioned, its localhost because that is how the ansible proxy adapter works. This works when the pipeline runs on Ubuntu-20.04, and does not work when the pipeline is run on ubuntu-latest (Ubuntu-22.04)

dendle commented 1 year ago

@mikhailkoliada I took time to prepare this in issue to help you find the cause of a potential problem with the new version of ubuntu that is rolling out. I think this is a real issue that will cause problems for your users.

dsouwar commented 1 year ago

I've had the exact same issue, and could only workaround it using the no_proxy setting so that the ansible provisioner uses the SSH connection created in the communicator and by pass the default ssh proxy.

actions / runner-images