[RFC] Rework initial password and certificates setup

[smortex]

Context

Initially, OpenSearch shipped with a bunch of hard-coded default user and passwords (i.e. admin:admin, anomalyadmin:anomalyadmin, kibanaserver:kibanaserver, kibanaro:kibanaro, logstash:logstash, readall:readall, snapshotrestore:snapshotrestore) and a hard-coded demo certification authority, server and client certificates.

Version 2.12 intended to improve the security posture of the project and now require to pass a OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable on first start or while installing. Concerns where raised when this was proposed and an extensive list of drawbacks of this approach where listed in this issue.

The current situation is that for running the server, the user SHALL provide a strong password for the admin user, making them think that their setup is not open to the wild. In fact, their setup still have half a dozen hard-coded user accounts with default passwords, and admin access is still possible using the demo user certificate of the kirk user which is hard-coded.

What this RFC wants to improve

1. Unbreak installation from packages by not requiring non-standard environment variables at installation time;
2. Allow the service to start without initial configuration (with limited functionality because no user account is configured);
3. Allow users to easily setup an insecure development environment;
4. Provide guidance to users to setup a production environment.

Proposal

At installation time, do not setup any hard-coded configuration: no internal users and passwords, no default certificates.

When the service is started, if the security plugin is installed but no certificate is found, generate a self-signed node certificate, use it to secure the communication to OpenSearch, but do not write it to disk. Restarting the service would generate a new certificate. With this absence of TLS material, no client certificate can be used to have admin access to OpenSearch, but the service is able to start, and can be operated the usual way providing HTTP Basic authentication when configured.

With no internal users by default, all request will be denied until some accounts are added.

The above behavior allows to install a package without the need to rely on non-standard environment variables, and if the service is started (some systems to this by default during installation and installation fail if the service does not start properly, e.g. Debian), it is not exposed to all abuses using default credentials: the service start, use generated TLS certificate, and all requests are denied.

At this stage, additional tooling would be welcome:

• A certificate generation tool that can generate an insecure demo configuration similar to the one currently installed at bootstrap by the security plugin: generate a self-signed CA, a service certificate and a client certificate. This is intended for development, and allows to use client certificates for authentication. But we do not install well-known keys and certificates as it is done today, but rather generate them. This allows multiple deployment using this insecure demo configuration to have distinct TLS material, and prevent such a deployment exposed on the internet to be accessible as an administrator using a well-known client certificate and key;
• A certificate verification tool that help to check that CA and certificates configured are valid (file permissions / Not before / Not after) and that certificates verify against the CA. The TLS material generated by the above tool would indeed generate valid configuration, but here I am more thinking about people rolling their own CA / server certificates / client certificates on multiple nodes, and allowing them to quickly check that their configuration is correct;
• An internal user accounts management tools that can be used to add internal users to the installation and manage them from outside OpenSearch in a non-interactive way;
• An internal user initial setup tool that use the above tool to help generate the bunch of internal accounts which are currently hard-coded, but prompt for a password for each user.

Expectations

With all the above, the setup of a development instance has a few more steps:

• Before:
  1. Install opensearch providing OPENSEARCH_INITIAL_ADMIN_PASSWORD
• After:
  1. Install opensearch;
  2. Run the certificate generation tool (optional);
  3. Run the internal user setup tool to create an administrator.

But the setup of a production instance has no pitfall:

• Before:
  1. Install opensearch providing OPENSEARCH_INITIAL_ADMIN_PASSWORD
  2. Setup TLS (mandatory to be secure, even if it's not explicit);
  3. Remove extra internal users, change hard-coded default passwords;
• After:
  1. Install opensearch;
  2. Setup TLS and run the certificate verification tool to detect errors early (recommended);
  3. Run the internal user management tool to add only required users.

The installation instructions for a development and a production site can therefore be merged in a single list:

1. Install OpenSearch
2. Setup TLS (optional) You can generate a self-signed certification chain suitable for development running xxx, or you can provide your own server and client certificates by storing them in xxx. Run xxx to verify your TLS configuration;
3. Setup internal users (optional) Use xxx to setup internal users.

[cwperks]

Thank you for the detailed RFC @smortex. I brought up similar concerns before as well: https://github.com/opensearch-project/security/issues/3632#issuecomment-1787985707

There are still many hardcoded secrets in the demo configuration and only changing the admin password can give a false sense of security, especially since the admin certificate is more powerful and can be used to meddle with system indices.

[DarshitChanpura]

+1 to the concerns raised.

Demo configuration tools is a long way to go from being completely secure, and changing the admin password was first step. There should be no default credentials or certificates provided. The OPENSEARCH_INITIAL_ADMIN_PASSWORD only comes into play when running demo configuration setup. If DISABLE_INSTALL_DEMO_CONFIG flag is passed the demo configuration is not installed and user's can directly modify the static config files to their need as required. The only thing demo installer does at this point is to modify admin password, setup security config in opensearch.yml and write the demo certificates to file-system. This tool can be separated into individual setups as mentioned in the issue.

[derek-ho]

[Triage] @smortex thank you for raising these concerns. +1 to all the concerns that you raised. I will mark this as triaged because we would love a contribution to make the demo configuration more usable! Maybe we can break this down into smaller tasks that can be done in parallel instead of one big issue.

[dblock]

At installation time, do not setup any hard-coded configuration: no internal users and passwords, no default certificates. When the service is started, if the security plugin is installed but no certificate is found, generate a self-signed node certificate, use it to secure the communication to OpenSearch, but do not write it to disk. With no internal users by default, all request will be denied until some accounts are added.

I like this very much.

[peterzhuamazon]

Add some of my previous suggestions of the password handle here:

1. Allow generating a random password from the start if user did not provide password, and find a reasonable way to deliver it to user. Jenkins is doing similar things like this.
2. Move the password check/gen process on deb/rpm from pkg installation to service startup.

Thanks.

[smortex]

Allow generating a random password from the start if user did not provide password, and find a reasonable way to deliver it to user. Jenkins is doing similar things like this.

There are multiple drawbacks to this, but I did not gave much context about the reasoning of why not providing any internal accounts, so please allow me to do so here:

1. Auto-generating a password is trivial, but securely transmitting it to the administrator is more problematic: we can't just output the password to stdout/stderr because it would end-up written to disk on remote logging system. We can write a password to a temporary file and output the name of this file, but it only move the problem because the filesystem might not be trustworthy and snapshots can even make recovering this unexpectedly. Also it is less practical;
2. We do not have a single password, but 7 with the default config. So we should not generate a single one, but 7 if we want to be consistent. That's a lot of information appearing out of the blue if this is auto-generated and not caused be some user interaction;
3. If you do not intend to use the internal users at all, having a list of passwords for users that will not exist is bogus.

My proposal is to not ship default internal user accounts. If at startup the internalusers database is loaded and is found to be empty, a warning is output telling the user about the situation and recommending to either not use this authentication backend or setup internal user accounts using the internal user initial setup tool mentioned above to bootstrap the configuration with system user accounts or the internal user accounts management tools to customize which accounts are managed.

Move the password check/gen process on deb/rpm from pkg installation to service startup.

Generation being out of scope for the reason above, the check part of course have to happen before starting the service. Installation should really just install files, and not attempt to generate anythings unless that does not require any user interaction (including providing but also reading a password). This was definitively a bad move.

Regarding checks, this is the goal of my proposal: the certificate verification tool can be used as a pre-start check for the service, and it allows administrators to use the same tool to check the files they are working on when setting up TLS for the first time or when renewing certificates.

[nibix]

That's a great proposal! I just came across it (kinda late), and have one further question:

If a node/cluster is initially spun up with self signed certs and no accounts - how does the internal user accounts management tool authenticate itself against the node/cluster? Just by locality?