Consul Connect CA: can't change provider

[pkrolikowski]

Hi all,

I'm getting error while trying to change Connect CA provider. Here are steps I've taken:

$ consul version
Consul v1.8.2
Revision ba7d9435e

1. enable Connect with Vault as CA provider:

   $ consul connect ca get-config
   {
   "Provider": "vault",
   "Config": {
       "Address": "http://localhost:8200",
       "IntermediateCertTTL": "8760h0m0s",
       "IntermediatePKIPath": "consul-connect-intermediate",
       "LeafCertTTL": "74h",
       "RootPKIPath": "consul-connect-root",
       "RotationPeriod": "24h",
       "Token": "5a6a08b4-e54c-38ee-acf0-387dac8fxxxx"
   },
   "State": null,
   "CreateIndex": 236021312,
   "ModifyIndex": 251656879
   }

2. Try to change provider to consul:

• Config to load (taken from working dev environment)

  $ cat connect_consul_ca.json 
  {
  "Provider": "consul",
  "Config": {
      "IntermediateCertTTL": "8760h",
      "LeafCertTTL": "72h",
      "RotationPeriod": "2160h"
  }
  }

• Try to upload config

  
  $ connect ca set-config -config-file=connect_consul_ca.json
  Error setting CA configuration: Unexpected response code: 500 (rpc error making call: error having Vault cross-sign cert: Error making API request.

URL: PUT http://localhost:8200/v1/consul-connect-root/root/sign-self-issued Code: 500. Errors:

• 1 error occurred:
  • error signing self-issued certificate: x509: requested SignatureAlgorithm does not match private key type

)

I tried to disable connect (remove connect stanza and rolling restart all servers) and enable it (provider set to `consul`) again but no luck, `consul connect ca get-config` still shows Vault as CA.

[pkrolikowski]

@jsosulska any thoughts on that?

[rboyer]

If you are knowingly trying to switch between rsa and ec when switching from consul to vault providers then you will run into this open issue in Vault: https://github.com/hashicorp/vault/issues/7709

I tried to disable connect (remove connect stanza and rolling restart all servers) and enable it (provider set to consul) again but no luck, consul connect ca get-config still shows Vault as CA.

Disabling connect does not delete the persisted CA roots in the data directory, it just disables the code that interacts with them. There is currently no way to fully reset the stored CA information outside of the set-config workflow.

Possible solution

If you left everything to the defaults, that means that both the consul and vault sides are dealing with ec keys and the error shouldn't be happening. I have a suspicion that something strange is happening, but let's see if you can bypass it. If the below works out for you then that'll provide a data point in favor of my unproven theory.

What we're going to do is reconfigure the Connect CA to use the builtin consul provider with a user-provided ec key.

First you'll need to generate a key directly and only keep the private key part:

private_ec_key="$(openssl ecparam  -name prime256v1 -genkey \
    | awk '/BEGIN EC PRIVATE KEY/,/END EC PRIVATE KEY/' )"

## Example contents:
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIESdAXC52Ec/ix0DL9sbbCVneK+0U+oFth+5hXe1K/ynoAoGCCqGSM49
AwEHoUQDQgAE0vd7DzpPfNyKd4HbGF76PhZryA+Ktmzg1vcSdec0jIfDTU/Rwm4B
8MSzpjOYyroFnIwgUXWtfPuDTvkpNtWr7w==
-----END EC PRIVATE KEY-----

If you pass that through a quick jq command we can get it encoded for stuffing into json:

private_ec_key_json="$(echo "${private_ec_key}" | jq -Rs . )"

## Example contents:
"-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIESdAXC52Ec/ix0DL9sbbCVneK+0U+oFth+5hXe1K/ynoAoGCCqGSM49\nAwEHoUQDQgAE0vd7DzpPfNyKd4HbGF76PhZryA+Ktmzg1vcSdec0jIfDTU/Rwm4B\n8MSzpjOYyroFnIwgUXWtfPuDTvkpNtWr7w==\n-----END EC PRIVATE KEY-----\n"

Now use that to generate your CA config snippet:

cat > connect_consul_ca.json <<EOF
{
  "Provider": "consul",
  "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RotationPeriod": "2160h",
    "PrivateKey": ${private_ec_key_json},
    "PrivateKeyType": "ec",
    "PrivateKeyBits": 256
  }
}
EOF

## Example contents:
{
  "Provider": "consul",
  "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RotationPeriod": "2160h",
    "PrivateKey": "-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIESdAXC52Ec/ix0DL9sbbCVneK+0U+oFth+5hXe1K/ynoAoGCCqGSM49\nAwEHoUQDQgAE0vd7DzpPfNyKd4HbGF76PhZryA+Ktmzg1vcSdec0jIfDTU/Rwm4B\n8MSzpjOYyroFnIwgUXWtfPuDTvkpNtWr7w==\n-----END EC PRIVATE KEY-----\n",
    "PrivateKeyType": "ec",
    "PrivateKeyBits": 256
  }
}

And use that to reconfigure:

consul connect ca set-config -config-file=connect_consul_ca.json

[pkrolikowski]

There is currently no way to fully reset the stored CA information outside of the set-config workflow.

Yeah, that's exactly what I wanted to do.

Thanks @rboyer, I'll try your solution :+1:

[pkrolikowski]

@rboyer unfortunately I'm still getting cert error:

Error setting CA configuration: Unexpected response code: 500 (rpc error making call: root certificate is expired)

BTW. I upgraded consul to v1.8.6, Vault is v1.2.3

[DejfCold]

There is currently no way to fully reset the stored CA information outside of the set-config workflow.

@rboyer Is there a plan to make it happen? Any related issue I could follow?

[ashwinkupatkar]

I am facing a similar issue. However, I haven't chnaged the provider. It is still consul. All I am doing is rotating the CA cert for the 2nd time and facing the issue as below: rpc error making call: error generating CA certificate: x509: requested SignatureAlgorithm does not match private key type

[ghost]

Same issue @ashwinkupatkar is facing here. Whatever key/key+cert combination I try to upload via set-config, I get the same error message:

rpc error making call: error generating CA certificate: x509: requested SignatureAlgorithm does not match private key type

I'd so much appreciate endpoints to manage data, like deletion of CA configurations, which might effectively allow to reset everything.