Alpha - Conjur signs certificates using a configured CA

[jvanderhoof]

Status

• Completion ETA: Alpha Version Complete (August 31, 2018)

Context

PKI, or Public Key Infrastructure is a mechanism to enable organizations to authenticate and secure communication between nodes in a network. Conjur will operate as a Certificate Authority, signing and managing certificates for hosts managed by Conjur.

Most PKI systems do not lend themselves to automation. The primary objective of this initiative is to enable organizations to leverage PKI practices at scale, in high automation environments.

Value

What is the benefit of Conjur acting like a CA?

• Allows an organization to leverage mutual TLS to provide application to application authentication using tools like Nginx and Envoy.
• Eliminates the need for engineers to create application-to-application authentication systems, allowing them to focus on shipping features.
• Provides application to application connection transparency (through mutual TLS), which allows a security team to track how applications connect to each other.

Feature Demo

To demonstrate Conjur acting as a certificate authority, we'll demonstrate the following scenarios:

• Issue a request (using Curl) to a server protected with mutual TLS.
• Connect to a simple website using a web browser configured with mTLS (potentially)

Setup

A best practice is to utilize an intermediate certificate to performing signing. This allows the root certificate to be kept securely, preferably in an air gapped environment. We will need to include the intermediate private key

Policy:

- !policy 
  id: conjur/<service-id>/ca
  body:
    # Signed certificates will be valid for up to a year
    - !webservice
        annotations:
          ca/private-key: ops/ca/private-key
          ca/private-key-password: ops/ca/private-key-password # If the import PEM key is encrypted
          ca/certificate-chain: ops/ca/private-key-chain
          ca/max_ttl: P1M

    - !group clients

    # Allow hosts in the `clients` group to be signed 
    - !permit
      role: !group clients
      privilege: [ sign ]
      resource: !webservice

Multiple certificate authorities can be configured by creating multiple CA services. This approach further simplifies the process of replacing intermediate certificate (due to breach or expiration). A new CA can be created, then remote nodes refresh their certificates using the new CA.

Multiple CAs can also be utilized to separate departments or divisions within an organization. Each certificate chain needs to include the appropriate certificates. We leave this as an exercise for the user.

Enable Host Signing

With our CA setup, let's look at a policy to enable application servers to create signed certificates:

- !policy 
  id: my-app
  body:
    - !layer

    - !host-factory
    layer: [ !layer ]

We can load my-app into the staging namespace with:

$ conjur policy load staging my-app.yml

Assuming we've created a staging CA (conjur/staging/ca), we can add our staging/my-app layer as follows:

# entitlements.yml
- !grant:
  member: !layer staging/my-app
  role: !group conjur/staging/ca/clients

$ conjur policy load root entitlements.yml

Signing a Host CSR

Any host that has a Conjur identity and signing permission on the CA can have its CSR signed.

Below is the API endpoint specifications:

API Specification

POST /ca/\/\/sign

Headers

Field	Description	Example
Authorization	Conjur access token	Token token="eyJkYX…Rhb="
Accept	Content type of response	Either application/json or application/x-pem-file is accepted. application/json is the default

Request

Parameter	Description
csr	(required) PEM encoded CSR file to be signed
ttl	(optional) The length of time the signed certificate should be valid for. TTL duration is in ISO8601 duration format. If TTL is greater then the configured CA TTL, the CA TTL will be used.

Response Codes

Code	Description
201	Returns the signed certificate
404	CA Service with given ID does not exist
403	Authenticated role is not a Host. Host does not have sign privilege for CA Service. Authenticated host does not match that of the CSR Common Name (CN).

Response (201)

# Headers
Content-Type: application/x-pem-file or application/json

# Body
## application/x-pem-file
PEM encoded certificate

## application/json
{
  "certificate": "...PEM encoded certificate..."
}

Example

$ curl --request POST \
        --d "csr=@/path/to/domain.csr" \
        --d "ttl=P1W" \
        --header "Authorization: Token token=\"$token\"" \
     https://conjur.myorg.com/ca/<service-id>/\<account>/hosts/<host-id>/sign

201
{
   "certifiate": " -----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----\n"
}

  $ curl --request POST \
          --d "csr=@/path/to/domain.csr" \
          --d "ttl=P1W" \
          --header "Accept: application/x-pem-file" \
          --header "Authorization: Token token=\"$token\"" \
       https://conjur.myorg.com/ca/<service-id>/\<account>/hosts/<host-id>/sign

201
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

The above request performs the following actions:

• Validates the CSR CN matches the host id (host1.my-super-application.staging.mycompany.com)
• Signs the CSR with the given TTL (ttl=P1W - optional, without ttl, certificate is valid for one month)
• Returns a PEM encoded certificate

Alpha Release

Stories

• [x] A CI environment exists for verifying PKI capabilities (parent of #682)
• [x] A PKI CA service is configured in policy (parent of #683)
• [x] A Mutual TLS example exists for Conjur CA signing (parent of https://github.com/conjurdemos/misc-util/issues/5)
• [x] PKI host certificates are issued through the Conjur API (parent of #684)
• [x] A CA service can be configured with either an encrypted or unencrypted CA private key (parent of #706)

Release Checklist

Technical

• [x] Feature can be demonstrated to a customer with a single use case: Demo
• [x] Basic test suite implemented

Supporting

• [x] Feature/Product has a name
• [x] Developer level documentation (Readme) has been completed for the feature: Developer Docs
• [ ] Security team has reviewed and approved the approach

Beta Release

Stories

• [ ] A CA service uses the authentication host is as the common name (#705)
• [ ] A CA service issues certificates with a SPIFFE compliant ID (#707)
• [ ] Conjur CA Services are fully documented (#688)
• [ ] CA service is refactored to increase OpenSSL re-use with Kubernetes authenticator (#725)

Release Checklist

Technical

• [ ] Majority of feature edge cases have been implemented
• [ ] Full integration test suite implemented

Supporting

• [ ] All new API endpoints have been documented
• [ ] Feature has been fully documented on the appropriate documentation site(s)
• [ ] Sales slide for feature has been created
• [ ] Marketing has information required for PR

GA Release

Stories

Release Checklist

Technical

• [ ] All use cases from the design document have been implemented
• [ ] Feature has been load-tested at expected production loads

  Supporting

• [ ] Blog post has been written and published
• [ ] Tutorial examples have been published
• [ ] Demo video is available
• [ ] Training session has been run for relevant PS and SEs
• [ ] Confluence page exists with supporting resources

Candidate stories

• ~A CA service can return the result as JSON, DER, or PEM format~(Included in #684)
• ~A CA service uses the host name from the authentication host, rather than from CSR~
• A CA service can accept a wider set up input parameters (e.g. common_name, alt_names)
• ~A CA service can use either an encrypted or unencrypted CA private key~
• A CA service can restrict signing by annotations on a host (e.g. ca/common_name, ca/allowed_alt_names)
• A CA service can issue SPIFFE compliant certificates
  • Include SPIFFE ID URI subject alternative name
• A CA service can be configured to sign certificates other than for the requesting host (Lemur use case)
• The Ruby API client supports certificate signing
• The Go API client supports certificate signing
• The .Net API client supports certificate signing
• Cron based example of how to rotate certificates exists
• Ansible playbook exists to:
  • Create private key & CSR
  • Sign CSR using Conjur

[kgilpin]

My first comment is: why combine the host creation via host factory and the certificate issuance into one operation? Instead, provide a route for any authenticated and authorized host to obtain a certificate for its host id.

[apotterri]

This is a great start describing a pretty interesting feature.

Some comments:

A best practice is to utilize an intermediate certificate to performing signing. This allows the root certificate to be kept securely, preferably in an air gapped environment. We will need to include the intermediate private certificate

The last line should say "intermediate private key", correct?

I think it would be helpful to have the new API entrypoints called out and described explicitly, maybe in the Workflow section. Only having them buried in examples make it harder to see what's actually going to be added.

In this block:

$ curl --request POST \
        -f "csr=@/path/to/domain.csr" \
        -f "ttl=P6M" \
        --header "Authorization: Token token=\"$host_factory_token\"" \
     https://conjur.myorg.com/ca/<service-id>/host_factories/hosts

the switches to curl should be -F, not -f. It might be better to use the long switch --form, to avoid confusion.

In the description after, it says

Signs the CSR (removing the CSR upon completion) with an expiration date six months from the request (ttl=P6M - optional, without ttl, certificate is valid for one year)

How does the CSR get removed?

Is "rotation" just what happens when a host with an existing cert gets a new CSR signed? Put another way, is there a difference between the first time a host sends a CSR, and subsequent times?

[kgilpin]

Providing a CRL makes the CA harder to implement because it requires a writeable data store to store all the revoked certificates. For example, a follower will not be able to perform the CA revoke function.

It is not the case that the need for a CRL can be all but eliminated by using short lived certificates? A CRL seems like something which would be used by internet facing machines which are using long-lived certificates. I would expect that to be the typical use case for this feature.

[apotterri]

Hope those comments help. All in all, I find that feature issues aren't very conducive to collaboration. Is there some other approach we could use?

[kgilpin]

@apotterri Often I see google docs being used for design discussions on public repos.

An alternative might be to open a branch for the purposes of fleshing out the design with documentation and interfaces.

[kgilpin]

@apotterri Normally for Rotation, the older but still unexpired certificate can be used as the credential to obtain the new certificate. That’s the main difference between rotating the certificate and obtaining the initial certificate (which would use machine identity credentials).

[kgilpin]

In summary, my recommendations are to: drop the CRL feature (for now at least), decouple the CA from host factory, and allow an existing certificate to be used as the credential for rotation.

[apotterri]

Yeah, a public Google doc seems like it would be a step up. If that's not possible, I guess a doc in the repo that can be branched would work ok.

Thanks for the clarification about rotation. As described, it didn't seem like a separate feature, so I was confused about why the term was used.

[jvanderhoof]

Thanks for the feedback @apotterri & @kgilpin. For the public development of features, GDocs probably make a lot more sense...

[jonahx]

@apotterri I'm curious what you don't like. I much prefer this method to using google docs personally...

[dividedmind]

Authentication shouldn't just follow from sending a certificate; certificates are public. Instead it should use standard client TLS authentication. That said, I don't like being able to authenticate with a cert at all, especially if they're long-lived. That also requires having revocation strategy (though unlike with using off-line authentication by third party, authenticating on-line with Conjur like this at least makes sure the revocation list is always available, since presumably Conjur knows it).

Seconding what @kgilpin said, I'd really rather see short-lived certs without revocation, not unlike our JWT tokens; in fact, exactly like our JWT tokens, except with different format and a choice of authorities. These certs can then be used for third party authentication (again, exactly like our tokens) using standard TLS mechanisms (unlike our tokens), and we can even support them as primary authentication method for Conjur requests (instead of our tokens).

But a lot depends on the exact use case we want to support. Just saying "we want CA" is an answer looking for a question. What is the question? In a way, Conjur is already a CA, except it doesn't use X.509 and is (intentionally) not very flexible.

[dividedmind]

@jonahx it'd be nice to be able to discuss inline.

[jonahx]

@dividedmind I agree with that. In fact I've thought for years now that a popular markdown based but inline commenting tool was oddly missing from the the tech landscape, and even thought of writing one. That said, I'd rather have markdown and plain text without inline editing than WYSIWIG with it. I just can't stand any kind of MS-Wordish interface, which ofc GDocs is.

[dividedmind]

@jonahx You mean like StackEdit?

In github we can have inline comments on a markdown by just editing an .md file in repository. Perhaps those design discussions should instead be PRs adding a doc to /doc? As a bonus we'd have a doc ready when we're finished.

[jonahx]

@dividedmind yes exactly like stackedit. though i've always imagined it as a widget that could be attached to any html page.

In github we can have inline comments on a markdown by just editing an .md file in repository. Perhaps those design discussions should instead be PRs adding a doc to /doc? As a bonus we'd have a doc ready when we're finished.

This sounds good to me. I just like having everything centralized in gh and being able to write with markdown.

[jonahx]

@jvanderhoof To respond to the post, now that I understand it fully, I'd like to echo Rafal's comments.

And agree it would be helpful to know what the high-level problem we're solving is.

[micahlee]

@jvanderhoof, regarding this policy snippet:

- !webservice
      annotations:
        ca/max_ttl: P1Y

I assume from this that the variables:

    - !variable intermediate-private-key

    # Certificate chain for the intermediate-private-key
    - !variable intermediate-certificate-chain

Are being linked to this CA service because of the shared policy ID. While this inference is probably fine as a a default, it would be nice to also allow the flexibility of specifying the variable to use. E.g.:

- !webservice
      annotations:
        ca/intermediate-private-key: ops/ca/private-key
        intermediate-certificate-chain: ops/ca/private-key-chain
        ca/max_ttl: P1Y

[jvanderhoof]

I updated the CA policy based on @micahlee's comments. Based on the feedback from @dividedmind, and @jonahx, I'm going to move the Cert based Authentication out of the scoped work. Initial work on a CA should give more context and insight.

[infamousjoeg]

So, taking this from a purely customer-based perspective, I'm loving all of this. I'm sure ya'll will work out the technical challenges... but you're going down the right path our customers are asking for.

[jtuttle]

In the k8s authenticator, it's an important point that the signed cert is injected into the correct host container out-of-band instead of returned in response to the CSR request. It was my understanding that this prevented someone from crafting a malicious CSR request to impersonate a host and using the cert they receive in the response to authenticate as that host. Is there something in this scheme that prevents such an attack?

[jvanderhoof]

Essentially we're building on the trust Conjur has through host identity. The CSR hostname needs to match host id as well.

[jtuttle]

@jvanderhoof But is there any way to verify that the host making the request corresponds to the host specified in the CSR? In k8s this is verifiable through the k8s API.

[apotterri]

@jonahx Just saw your comment now. I miss having threading on discussions (which would make this comment appear on your comment, instead of having to @ you from way down here).

I miss being able to have my comments tied to a particular line in the source material. Having to quote from the doc for context makes it harder to be clear about what I mean.

I miss having good versioning of the doc. For the feature issue in conjurinc/appliance I was working on, I considered creating a branch with a doc, and pushing changes to it. When I realized that was going to kick of a build each time I pushed, I abandoned that approach. That's less of an issue in this repo, so this would line up with @dividedmind's suggestion.

I don't have any particular desire to use a Google Doc (though I've found them to be much more workable than any time I've tried to share a Word doc).

[jvanderhoof]

@jtuttle: what about checking the incoming request IP and verifying the CSR hostname resolves to that IP?

[jtuttle]

@jvanderhoof We've had trouble with this in k8s due to hosts having services in front of them. We had to disable IP validation because we found that in some cases the IP of the request didn't match that of the host.

[jvanderhoof]

This feature wouldn't (at least initially) be intended for Kubernetes. We already have this with Authn-K8s. As I understand the need, it's for vanilla instances. To that point, there's definitely going to be some additional research and exploration we'll need to do to verify the host.

[kgilpin]

@jtuttle the host is presenting a Conjur access token and it can only send a CSR for the same host id as the one on the token.

This is functionality built on top of the trust established by the Conjur access token. Unlike authn-k8s, it’s not a means of establishing trust in the first place.

[kgilpin]

And IMO it only makes sense to do this for short lived certificates, because to implement long-lived certificates is a much harder problem. If someone wanted that I would recommend them to run something like the Lets Encrypt stack on-premise.

[jvanderhoof]

The use case that comes to mind is using this in conjunction with something like Envoy. The primary goal seems to be app to app authentication. In that scenario, I'd hope certificates were rotated regularly.

[rafis3]

• I think it would be great to integrate the intermediate CA with the authn-k8s CA so certificates issued for K8s would be CA signed rather than self signed only.

  Where Conjur can have multiple CAs, we'll need to think about how best to do this. One option would be to provide an optional Conjur intermediate certificate path. This intermediate certificate could then be used by the authenticator.

• Are we going to issue the CSR as SPIFFE compliant? That would be nice.

  We definitely should try to adhere to the SPIFFE framework where possible. With this current design, it would require Conjur host IDs follow the SPIFFE format. Alternatively, if we add CN to the signing request, the CN could be a SPIFFE compliant ID.

• Just want to make sure I'm not missing anything, in the two conjur CLI examples, the word conjur repeats twice. Is that on purpose?

  Fixed. Thanks for catching it!

• Part of this effort, are we going to add the generation of the CSR into a select Conjur API libraries? (Ruby, Go, Java, etc). If yes, then which API libraries?

  I wonder if it makes more sense to build this into a couple of CM tools: definitely Ansible, and maybe Puppet & Chef, as well as provide a bash script. Each of these should • Generate a new private key and CSR (if not present) • Authenticate with Conjur using the host identity • Send the CSR to Conjur for signing, place the resulting CRT on the file system

• Are we considering supporting a flow in which the client doesn't have to generate a CSR (to simplify the client side code). So the client would ask for a certificate and the generation would be done completely on the Conjur side.

  All the research I did on cert signing recommended staying away from this approach. It's critical that the private key stay private. The best way to do that is to keep it on the host where it was generated.

• Default expiration of 1 year to a certificate seems very long to me. If we intend this feature for short term certificates only, I think we should limit the max allowed TTL to the max length we feel comfortable with. Or at least set the default to a low recommended value. In addition, warn in the documentation about the security consequences of making it longer.

  Completely agree. I dropped the CA TTL down to one month and the host example down to one week. We'll need to document the importance of short TTLs. Rotation (potentially using Cron) would be good to discuss as well.

• In the authn-ca section, this was mentioned: "The host uses the CSR for subsequent requests to Conjur". Does that mean the certificate would be exchanged for an access token and then the rest of the flow is regular (which is giving the Conjur access token)? Or that subsequent requests would allow getting the certificate as well with mTLS? (as an alternative to the Conjur access token) I would think that we would rather go with the first option, to minimize our changes in the existing APIs, but I also just want to make sure I understand correctly the "subsequent requests" part.

  I yanked this for now (and pushed it in a potential future epic). I completely agree with you. Kevin and Rafal also had some concerns about using mutual TLS instead of a short lived token. I'd love to keep exploring what this flow might look like.

[TheSecMaven]

Is this available in Conjur for use today? Best I can tell this dropped off 2 years ago and never made it over the finish line.

This feature is in hashi vault, so would be HUGE to have it in Conjur :)

[micahlee]

Hey @mkkeffeler, thanks for asking!

This particular alpha functionality is in Conjur and DAP today and available to try. There are some basic docs around it here: https://github.com/cyberark/conjur/blob/master/design/CERTIFICATE_SIGNING.md. If you have questions around that, feel free to ask here, in a new GH issue, or on the community site. I'm happy to try and help get it working for you.

The focus of this alpha was on signing certificates for mutual TLS based on host identities in Conjur. We've had a few other prototypes around certificate signing use cases, but nothing else that has landed in a Conjur release yet.

The most recent work around certificate signing supported by Conjur is available in another repo here: https://github.com/infamousjoeg/cyberark-aam-pkiaas/. That may also be worth checking out.

I hope that helps!

[infamousjoeg]

Hey @mkkeffeler,

I am one of the 4 contributors to the PKI Service for CyberArk that was linked in the above reply. If you have any questions, please feel free to reach out to me through the "Discussions" section of that repository.

Currently, it's in an alpha state, as well... but is a bit more polished in the use cases and has been demonstrated to multiple orgs. We are working on merging a few branches into the master branch, such as a CLI, automated init and policy load, and Signed SSH Certificates.

Please leave us feedback and let us know if there's anything we can do to help!