ability to trust private pki

[thenewguy]

It would be handy if there was a method to add a root certificate to the bundle.

My particular case: Some 3rd party apps I am using rely on the requests package. I cannot modify them directly to trust the internal pki. I had to do the following:

pip install certifi

python
import certifi
certifi.where()
quit()

cat /path/to/trusted.pem >> /path/to/virtualenv/site-packages/certifi/cacert.pem

The problem with this is that updating the certifi package will overwrite the cacert.pem file and the app will break.

It would be much nicer if I could do something like the following at the top of my django settings.py file:

import certifi
certifi.add("/path/to/trusted.pem")

Then certifi would inspect the certificate to add. If it was already included in "/path/to/virtualenv/site-packages/certifi/cacert.pem", nothing would happen. Otherwise it would append the trusted certificate to the bundle.

[Lukasa]

That does sound like an interesting idea. The thing that concerns me about that approach is that it leads to implicit global state on the system: one application adding a cert to the trust bundle will affect any other part of the system that uses it, which is potentially quite alarming.

Realistically, what we want is an in-memory API for OpenSSL. That would allow us to handle this much more cleanly than with this proposal. Remind me @reaperhulk, is that a thing I can ever have?

[sigmavirus24]

So, in a world where we can take advantage of SSLContexts (and yes, I know that I am holding that up) can we not load more than one path for a certificate bundle? Isn't that what load_verify_paths (probably not the exact name) does? If so, could certifi accomodate that and just have a way to keep track of this?

As I understand @thenewguy's current proposal, we would open certifi's PEM file and append the "trusted" one to it. This isn't something we should do and it's certainly not something we will have permission to do.

Let's say that a system is provisioned for a user that includes certifi. If that user doesn't have sudo access and certifi was installed globally, that user will not have permissions to update the certificate bundle. So we need to accomodate that use case. And besides, even if the user does, there's nothing to stop the user from including a malicious root to allow them to MITM connections on that box. I don't think we want to enable that vector.

That said, having a way for a program to say (for it's own execution) "Add this to the list of trusted sources" makes sense to me I think, but that won't solve this use case. (Especially if you're using a library that uses requests but doesn't provide a way to specify a bundle.)

[thenewguy]

Well, if the code has permission to use the hypothetical certifi.add(...) it would also have permission to manually modify the bundle. If you are trying to stop malicious code, I think there is a deeper problem.

Also, isn't it best practice to be running inside of a virtualenv? I typically serve python applications in their own environment so I hadn't considered the global installation.

I suppose if you need to support global installs running multiple application code without sharing certificates it would be simple to read the bundle into memory and use a temporary file on disk for passing the information along to OpenSSL. Then a new temporary file could be created each time the bundle was modified. This would fulfil your application segregation requirement and probably be fairly simple to implement.

[Lukasa]

Also, isn't it best practice to be running inside of a virtualenv?

Yes. Sadly, security-critical infrastructure should not assume best-practices. =(

Temporary files on disk work, but I'm more interested in being able to build an in-memory PEM database. That'd be a good feature for certifi anyway, so if we can make OpenSSL shut up and take them I'd vastly prefer to take that path.

[thenewguy]

security-critical infrastructure should not assume best-practices

Good point.

[thenewguy]

Any updates on the feasibility of an in-memory PEM database?

[bretkikehara]

If we take a step back, instead of importing a private pki at runtime, it may be good to assume the private trusted CAs are located in the user's home like ~/.certifi/trusted.pem?

From a security standpoint, if an attacker can create the trusted.pem in the user's home, then there is probably a more systemic problem.

[thenewguy]

It would be much nicer to be able to add simple instructions like

simply run certify.trust_my_bundle("bundle-as-string") in your settings.py file
and this project will integrate with your private pki

than it would be to provide cross platform installation instructions for installing a cert bundle on the file system.

[Lukasa]

I'm really nervous about the idea of providing some global way to hook into certifi. Generally speaking, I think the better approach is the one I'm pursuing in certitude: provide functions that allow validation of certificates from the system trust root on all platforms, and then simply use it.

[thenewguy]

@Lukasa Using the system trust would work just fine for my purposes. But if the certifi bundle is local to the running application code, is there really any danger in allowing a way for the application to update the trust bundle used for the running process? If an attacker is able to execute code at the application level then he could simply bypass certificate validation right?

[Lukasa]

There's no danger to letting the application do it, but there's not really any utility in providing a function for it either. The user is quite capable of writing the small amount of glue code required. What I'm averse to is providing any form of machine-global state that does it.

[thenewguy]

@Lukasa There is utility. For example when the application runs on a PaaS that does not allow the user to adjust the global system trust bundle. And also for documenting the integration. But if you're working on a better approach that sounds awesome

[Lukasa]

What I mean when I say there is no utility is that the code to do this is about 4 lines: just append the certifi PEM bundle to the PKI bundle you already have. Having that be a utility function in this library is just a bit unnecessary I think. =)

[thenewguy]

@Lukasa Oh sure. Earlier in the thread it talks about creating an in-memory PEM database or possibly using an arbitrary temporary file in which case it seems a little more complicated to adjust.

[rmalleman]

I'd just like to throw in my support for this. I have an application that makes https connections to hundreds of different endpoints, some using obscure root certificates that aren't included in the mozilla bundle. Without the ability to add trusted root certs I'm stuck compiling together my own ca-bundle, or just making https connections without certificate validation to some endpoints. I'd really like to avoid both of these scenarios,

[calpaterson]

I think the upshot of this thread is that best option for a user with private PKI is for the user to edit cacert.pem after certifi is installed to add their private PKI. That is a pain.

[compuguy]

@calpaterson editing cacert.pem isn't easy if your building software that needs internal corporate certificates to work with internal sites. I'm thinking of just making an internal fork of python-certifi and just appending our certs to the chain (since the code I'm working with uses the python requests library) to connect to go.cd servers: https://github.com/greenmoss/gocd-parser

[calpaterson]

Agreed, having to edit cacert.pem is very difficult. Red Hat ship certifi RPMs that replace certifi's cacert.pem with a symlink to the system trust store:

https://admin.fedoraproject.org/pkgdb/package/rpms/python-certifi/

If you're using a Red Hat derivative that might be something that works for you (depending on your dependency isolation).

[compuguy]

@calpaterson This code is actually running on Ubuntu 14.04, so there isn't a package available for me to use.

Edit: After looking at the patch, I think I can use this to make a variant of python-certifi to point to the Ubuntu cert store vs the certifi pem, which would be less disruptive than what I was considering.

[xeor]

Any plans for adding support for this? A workaround for now would be setting REQUESTS_CA_BUNDLE (if requests is the problem... Something like: REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

[Lukasa]

Nope, no plans to do that. Certifi could be part of a solution that does this, but it will never do it itself: right now it does one very specific thing (provide the Mozilla root cert bundle) well.

[mikeatlas]

The success of the package now is a handicap. While the original purpose of the package was to supply a safe 1-off CA bundle, many packages have adopted this and as such, I can't easily instruct the packages to pick up an additional internal CA cert to be trusted. Modifying the packages / code to pick up 1-off is not a possible solution. A solution someone suggested was to call where() and then amend the PEM (link) but that will fail if certifi is re-imported again at runtime. That RHEL forked the package to pick up the system default bundle shows that its useful and safe to do so.

Basically, consumers of 3rd party packages have no easy way to get that code to pick up internal CA certs without forking every package using certifi and changing the code where needed to use an internal certificate, as you're suggesting. Disappointing - an environment variable to use would be a safe improvement, not a security risk. Take the RHEL package as proof.

[Lukasa]

Disappointing - an environment variable to use would be a safe improvement, not a security risk

Cool, sounds like you’ve found a way to make a hugely popular Python package: congratulations! Feel free to write a wrapper package that implements just this. It should be about 4 LoC: you’ll have more code in your setup.py than in your module.

If your theory is right it should be a no brainier for every package currently using certifi to move to it. I look forward to you proving your success. 😁👍

[xeor]

I can see why you are tired of this issue, but this problem is real, and really annoying. Certificate-trust is hard enough inside large organisations that uses their own CA.

Based on your comment 11th of august, this is not in the scope of this lib. Are you then suggesting another lib building on this, and it's own code, to provide such a feature? Since certifi is already so much used, I don't see this problem getting a solution anytime soon..

[Lukasa]

Based on your comment 11th of august, this is not in the scope of this lib. Are you then suggesting another lib building on this, and it's own code, to provide such a feature?

Yes.

This should be extremely simple to do. Certifi has a public interface that is one function long, that takes no arguments and that returns a string. This is about as simple a module to extend as you can imagine.

A wrapper module that matches this interface would be almost trivial to implement. Someone doing that would therefore let people have the option of using a more fully-featured solution where that is needed, while keeping this module exactly what it is.

Again, I should stress the goal of this module. Its only goal is to provide the Mozilla trust bundle in a form suitable for HTTPS. That is it. It does not promise to provide a custom trust store for specific use cases, or to be the cross-platform Swiss Army knife of TLS implementations. Those are good things to build, but they are not this. Certifi is one component in such a thing, and we built it. People should feel free to build on top of it.

[mikeatlas-r7]

@Lukasa I get the perverse part about making an open source tool that becomes popular - the more we demand of you the more your "gift" becomes a burden.

The thing is, configuring a CA bundle to a sensible system default is not a code-level problem as you suggest. Python doesn't seem to come with a sensible OS level CA bundle lookup path, which is presumably why you made this package and why so many others import it.

However, writing a wrapper is not feasible for me - the code in question happens to be running running in 3rd-party immutable Docker images. I'm generally only able to set run-time environment variables when the images are run as containers.

I don't expect you to be entirely sympathetic but the snarky emoji isn't warranted either. I'm not the first to struggle with the design choice of "doing one thing and doing it well" (your mantra here and that's fine). It's just that there's no escape hatch here - in my case, it's doing the wrong thing. Heck, I'm so downstream from this package that it shouldn't even matter, but this is where the root of my challenges with getting these containers to pickup internal CA bundles is rooted. I can't ask that other packages be recompiled nor have the images re-released.

[alex]

I'd suggest bringing this up with one of your other upstreams. As @Lukasa said, this module has an extremely narrow purpose, and your problem is better solved elsewhere in the stack.

[mikeatlas]

Got it. Not your problem, won't fix. Why not close the ticket?

[davidlday]

Late to the game, but for what it's worth I had to tackle this very problem at work - needing to add our internal certificates for trust. I created a new package (let's call it 'corpcertifi') that mimics certifi, included a simple build script that uses certifi.where() to find the base cacert.pem, copies it to its own module directory, concatenates our internal certificates to the end, checks it in to an internal git repo, and then publish it to our internal pypi repo for broad use. Still working on those last two pieces, but that has nothing to do with certifi.

Requests works just fine when setting the REQUESTS_CA_BUNDLE environment variable (within the Python code) to corpcertifi.where(). Something similar could be accomplished in an application's code if there were certs only relevant to that application.

I'd be reluctant to use a module that allows trusted certs to be altered altered at runtime. I'd much rather have a version controlled way of knowing which certs are trusted.

[mikeatlas-r7]

So, what I ended up doing was adding a new __init__.py which monkey patches certifi.where() to look for the common OpenSSL environment variable:

old_certifi_value = certifi.where()
def where():
    try:
        env_var = os.environ['SSL_CERT_FILE']
        if len(env_var) > 0 and os.path.exists(env_var):
            return env_var
        else:
            return old_certifi_value
    except Exception as ex:
        return old_certifi_value
certifi.where = where

It's ugly, but it works.

[jalaziz]

Can't believe it's been nearly 3 years with no movement to supporting private PKIs.

The solutions suggested above all seem to expect you to own the code. That's not always the case and it's a bit sad that projects like requests have to resort to handling it separately. Trusting internal PKI is hard enough as it is, it would be great if certifi could provide a consistent interface around this.

I understand the point of this project is simply to provide the Mozilla root certs and creating a new project that provides the ability to add private CAs make sense. However, getting projects that already use certifi to now use a new module seems like a big ask, especially when this project could naturally evolve in that direction.

The reality is requests is such a popular project that unless requests switches to this a new wrapper module, I can't imagine others would follow quickly or easily. Given this project's tagline is Trust Database for Humans, I was hoping for a bit more.

I am probably just introducing noise at this point, but it would be awesome if there truly was a Trust Database for Humans just as requests is HTTP for Humans.

[Lukasa]

Can't believe it's been nearly 3 years with no movement to supporting private PKIs.

Yes, it has been three years of people complaining on this issue instead of stepping up to solve the problem themselves.

I have minimal patience for being told that due to our third party users we have some duty to expand our feature set. We do not. If you want a library to support a custom trust store, take it up with that library.

[jalaziz]

@Lukasa That comment wasn't directed at you, but the community in general.

Why should libraries have to support a new custom trust store when there can be one trust store to rule them all? No one said that you have some duty to expand the feature set. I was simply saying the tagline of the project inspires hope for more.

However, since everything is a personal insult to you, I'm going to stop with this, it's sad to see such a promising project being maintained by someone with your attitude towards the community.

[Lukasa]

I didn't take this as an insult: I'm just saying no, and explaining that I am not interested in an argument that we have some kind of duty here. It has been made clear, multiple times, that this functionality is out of scope for certifi. That is the final word on the matter.

I appreciate that you would like to make a plea to the community to build some software to solve your problem. That's a reasonable desire. The venue to do so is a blog post, not a three-year old issue on a bug tracker for a repo with 250 GitHub stars. The "community" is not reading this issue tracker. When you comment here, the only person who is guaranteed to respond to you is me, so forgive me for assuming your comment was aimed at me.

[mikeatlas]

Every other cert store bundle in browsers, operating systems and other applications support adding private certs. Seems like keeping this ticket open is just for sick burns and insults. Why not just close it if you’ve said your “final say”?

[Lukasa]

Ok. :smile: