Failed to parse key file

[domdip]

From some preliminary Google searches this is possibly due to ssh-agent (not confident in that though). I'm trying this on OS X.

* module.airflow-cluster.aws_instance.airflow_webserver: Failed to parse key file "-----BEGIN OPENSSH PRIVATE KEY-----\n*redacted*\n-----END OPENSSH PRIVATE KEY-----\n": ssh: cannot decode encrypted private keys
* module.airflow-cluster.aws_instance.airflow_scheduler: Failed to parse key file "-----BEGIN OPENSSH PRIVATE KEY-----\n*redacted*\n-----END OPENSSH PRIVATE KEY-----\n": ssh: cannot decode encrypted private keys
* module.airflow-cluster.aws_instance.airflow_worker: Failed to parse key file "-----BEGIN OPENSSH PRIVATE KEY-----\n*redacted*\n-----END OPENSSH PRIVATE KEY-----\n": ssh: cannot decode encrypted private keys

Haven't tried to work around it yet but let me know if there's more info I can supply.

[edbizarro]

Hi @domdip the module assumes you have a private and public key at ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub but that's configurable through private_key_path and public_key_path if you have different paths from theses you need to update them in your config.

Can you provide your configuration?

Thanks

Edit: your keys are password protected?

[domdip]

Hi @edbizarro I have public and private keys at those locations (and am using the default).

My keys are password protected. ssh-agent uses the keychain - in normal use I don't need to enter the passphrase.

~/.ssh$ ls id_rsa*
id_rsa      id_rsa.pub

~/.ssh$ cat config
Host *
   AddKeysToAgent yes
   UseKeychain yes
   IdentityFile ~/.ssh/id_rsa

~/.ssh$ ssh -T git@github.com
Hi domdip! You've successfully authenticated, but GitHub does not provide shell access.

module:

module "airflow-cluster" {
  # REQUIRED
  source              = "powerdatahub/airflow/aws"
  aws_region          = "us-east-1"
  key_name            = "airflow-keypair"
  cluster_name        = "my-airflow"
  cluster_stage       = "dev" # Default is 'dev'
  db_password         = *redacted*
  fernet_key          = *redacted*

  # OPTIONALS
  load_example_dags   = true
  load_default_conns  = false
  rbac                = true
  admin_name          = *redacted*
  admin_lastname      = *redacted*
  admin_email         = *redacted*
  admin_username      = "admin"                           # Only if rbac is true
  admin_password      = *redacted*
}

[edbizarro]

Before i dig into this problem, can you update the module and test again? I made a small change in the connection section to not use agent

[domdip]

Encountering the same problem after terraform get -update (then destroy / apply).

I can create another pair that doesn't have a passphrase to work around this for now. Thanks for looking into it, happy to try something else if it'll help.

[domdip]

Per this link, specifying the private key may prevent it from using the agent. (Would probably need to look at terraform code to be sure. They've changed this logic around over the years it seems.) I think in my situation I'd need agent=true and no private key specified for it to work properly.

https://stackoverflow.com/questions/43197751/does-terraform-support-ssh-password-protected-key

[edbizarro]

I can create another pair that doesn't have a passphrase to work around this for now. Thanks for looking into it, happy to try something else if it'll help.

Can you confirm that with keys without password worked?

[domdip]

Keys without passphrase got me to the 'stat' error. That may or may not be an SSH issue. I've been trying to find a minimal case that works with one of these providers.

I've managed to get past the stat error but now I'm seeing a timeout (after 5 minutes). Digging into that, the security group attached is not being created properly. You use module 'terraform-aws-modules/security-group/aws'. I'm not sure if I pulled a buggy version of that, but your configuration looks fine yet it's creating something with no ingress or egress rules.

Until I get past these I'm afraid I can't tell whether SSH provisioner is configured properly. But I do think (based on comments elsewhere) that 'agent = true' was being ignored when a private ssh key was supplied (which will cause problems when that private key has a passphrase).

[edbizarro]

https://github.com/PowerDataHub/terraform-aws-airflow/issues/10#issuecomment-482409785

[domdip]

Confirmed I am able to get it going in latest (with agent = false and passing private key that does not need passphrase).

It may have worked with 'agent = true' (didn't test against that). As mentioned, I think that was being ignored in favor of private key.