[FEATURE] Add hot reload of certificates using file system

[willyborankin]

Is your feature request related to a problem? Observed here https://github.com/opensearch-project/security/issues/4144 and https://github.com/opensearch-project/security/issues/1763

Currently existing endpoint PUT /_plugins/_security/api/ssl/{type}/reloadcerts reloads SSL certificates only on the corrdinator node but not for the whole cluster.

What solution would you like? One of the possible solutions to fix it is implement possibility to hot reload SSL certificates from the files system automatically for each node. How it was done e.g in ES.

What alternatives have you considered? If we want to use and endpoint for it. The reload must be done via master node.

[scrawfor99]

[Triage] Hi @willyborankin thank you for filing this issue. This seems like a good request and something that would be helpful for lots of people.

[shikharj05]

Thanks for filing this! Couple of thoughts-

Currently existing endpoint PUT /_plugins/_security/api/ssl/{type}/reloadcerts reloads SSL certificates only on the corrdinator node but not for the whole cluster.

Currently, the API reloads the certificate on the node it is called, right?

One of the possible solutions to fix it is implement possibility to hot reload SSL certificates from the files system automatically for each node. How it was done e.g in ES.

I would like to understand more on this, when reloadcerts is called, the node can reload files from disk here

Do you wish to propose that API call to coordinator node should trigger reloads on all nodes? I see couple of challenges with this approach, for e.g. for an n-node cluster, this would require all 'n' nodes to have the latest certificate files ready?

[willyborankin]

Thanks for filing this! Couple of thoughts-

Currently existing endpoint PUT /_plugins/_security/api/ssl/{type}/reloadcerts reloads SSL certificates only on the corrdinator node but not for the whole cluster.

Currently, the API reloads the certificate on the node it is called, right?

Yes

One of the possible solutions to fix it is implement possibility to hot reload SSL certificates from the files system automatically for each node. How it was done e.g in ES.

I would like to understand more on this, when reloadcerts is called, the node can reload files from disk here

Yes

Do you wish to propose that API call to coordinator node should trigger reloads on all nodes? I see couple of challenges with this approach, for e.g. for an n-node cluster, this would require all 'n' nodes to have the latest certificate files ready?

Yes and it is up to the end user. We definitely need describe in details in our documentations what they should expect in this case. But I would like to fix existing endpoint and will do it later.

Cases in which such functionality would be useful:

• for full SSL mode we do not provide endpoints to manage cluster users including reload certs (https://github.com/opensearch-project/security/issues/3466)
• in case of CCR and CCS if you integrate different clusters with different certs both cluster should use certs to decrypt messages on the transport level, so hot reload without full restart of both clusters is a good idea
• not all users want to use REST API to reload certs, due to security reasons or they do not control SSL certificates on the transport/http level, some end users could use Terraform provider which upload files on nodes and do not use REST API
• such requests were done in the past e.g. https://github.com/opensearch-project/security/issues/172
• help to migrate from ES, since it supports reload of SSL certs this way, so end users already know what expect and already have automation for it

And one more important thing. If we give possibility to end users reload SSL certs via endpoint it is a potential security hole and it is a serious problem, since any vulnerability in our REST API could lead to the serious problems for users. If you take a look precisely for other projects e.g. HAProxy you will find out that they all control the file system but do not open access to reload certs via endpoints or something.

[shikharj05]

Thank you for adding the details! A few follow-ups-

Yes and it is up to the end user. We definitely need describe in details in our documentations what they should expect in this case. But I would like to fix existing endpoint and will do it later.

Agreed that existing endpoint should not be modified with this feature, it can be done later.

for full SSL mode we do not provide endpoints to manage cluster users including reload certs (https://github.com/opensearch-project/security/issues/3466)

+1, Security plugin should support a way to manage and reload certificates for SSL mode.

And one more important thing. If we give possibility to end users reload SSL certs via endpoint it is a potential security hole and it is a serious problem, since any vulnerability in our REST API could lead to the serious problems for users. If you take a look precisely for other projects e.g. HAProxy you will find out that they all control the file system but do not open access to reload certs via endpoints or something.

Discussing security of the the REST API endpoint is a great point, however, it also offers some ease of use today v/s how HAProxy does it using local sockets and access to the host is required to run these.

That being said, I do see benefits of adding an endpoint-less approach, file-based approach will be a good feature addition!