Automatic updates at run time

[g-andrade]

This is only a proof of concept for now, as I would like to get some input on whether this is actually feasible.

Main points:

• an updater process is launched under the app supervisor
• this process will regularly download, from a HTTPS URL, updated versions of the CA bundle onto a file in an user cache OS directory
• in case of success, the certifi module will then be recompiled so that it loads and exposes the new bundle while still enjoying the performance benefits brought out by the ct_expand parse transform.

Tricky bits:

• because the download must be secure itself, ssl_verify_fun must be imported as a dependency and the usual CA verification boilerplate enforced;
• for this, the latest loaded CA list is used; there's the unlikely possibility that a very outdated version of certifi tries to update using a HTTPS URL whose certificate was signed by a CA still not listed on the bundled list, and therefore be put in a chicken-and-egg sort of situation;
• recompiling modules in runtime with parse transforms is a bit obscene

To do (in case this goes ahead):

• conditional HTTPS requests (no sense in downloading the same bundle over and over)
• gracefully dealing with expectable failures

[ericmj]

I don't think it should do runtime updates by default.

[g-andrade]

👍 that's reasonable.

[benoitc]

I will have a closer look on the source code but at a first glance, why not building dynamically the module at runtime and switch the version instead of introducing a new layer?

[g-andrade]

why not building dynamically the module at runtime and switch the version instead of introducing a new layer?

You mean doing a new build of an updated version of certifi and then deploying that as a hot code reload?

If that's it, it would work, but still require manual intervention. What I seek is something that can update itself so that systems can be left running for months / years without risking sudden failure because some important external service based on SSL starts serving a certificate that's signed by a new CA that certifi doesn't know about.

[benoitc]

mm no I was thinking compile the beam and replace it directly in memory using merl for example, there is no real need to redeploy in such a case.

[benoitc]

@g-andrade just pushing the idea anyway. I'm fine with ETS but using a beam should make it more efficient for something called that often. Thoughts?

[g-andrade]

Oh, but wait. The way I did it, the module is recompiled and the ct_expand parse transform re-applied.

[g-andrade]

The ETS table is used only for more easily sourcing the CA bundle (either from priv dir or from memory.)

[g-andrade]

Any news?

[benoitc]

@g-andrade needs to revisit that patch but made a quick review last week:

1. we should probably make the validation of the downloaded file stronger. I'm thinking to use a public key added during the server compilation.
2. An idea to improve it was to put the result directly in the SSL cache. I'm not sure yet if it's possible, but that would reduce the memory usage

Otherwise I quite like the idea. This can be added for the next release next week IMO :)

[benoitc]

@g-andrade do you think you can make such changes? that would be helpul. I will take care of it anyway :)

[g-andrade]

I might find the time for it on the weekend, but no promises.

[g-andrade]

I might find the time for it on the weekend, but no promises.

Well, it looks like I didn't.

Closing this - I don't think I'll ever resume work on it. But I'll keep the branch around, in case someone wishes to pick it up.