[Proposal] Finalize release steps for securing installation

[setiah]

Is your feature request related to a problem? Please describe

Identify breaking changes w.r.t. developer or user experience and finalize the release process for the new setup changes to secure installation.

Describe the solution you'd like

Background

The solution is based on the premise

• Only non-breaking changes should be part of minor version release.
• Breaking changes (w.r.t. user experience, setup experience), should be part of next major release.

Let's take a look at what changes and identify how should it be released.

PR 1 - New certificate setup tool Deprecates demo certificates and replaces them with freshly generated self-signed certificates.

1. The certificates are installed in a new certs/ folder under config to keep the config/ cleaner.
2. If users have hardcoded the demo certs installed in any of their clients (which is a possibility because the demo certs haven’t changed since the odfe launch), those clients won’t connect to OpenSearch unless those demo certs are replaced with new self-signed certs generated on the setup.

Since both 1 and 2 break existing assumptions in some ways, this PR qualifies as a major version change and should be a candidate for 2.0

PR 2 - Replace demo certs with new certificate setup tool in Docker distribution Removes pre-installed demo certificates in the docker image. Instead, packages the Certs tool in the docker distribution to generate certificates on the fly when docker container runs for the first time. Also changes the docker-compose file.

1. If users have hardcoded demo certs installed on client side, they won’t work and would need to be replaced with new certificates generated by the setup.
2. docker-compose.yml changed.

Both 1 and 2 are breaking changes for users, hence this PR too qualifies as a major version change candidate.

PR 3 - Replace admin:admin default credentials This change is wip (cc: @ryanbogan ). The plan is to remove the weak default credentials and provide an alternative mechanism to setup strong passwords in the new out of box experience Although this change has nothing to do with the install_demo_configurations.sh tool, but it is part of the new out of box experience and should be released alongside. Also, this is a breaking change like the other two described above, as the users would no longer be able to hit the APIs with weak admin:admin password, and would need to setup passwords differently.

Proposal

The proposal is to mark the existing install_demo_configurations.sh tool as deprecated in 2.0, while also adding the new replacement tool to the distribution, which should be used to setup certificates and other configurations related to security. The documentation website should be updated to communicate the deprecation of the existing tool in favor of the new tool. The docker distribution should start using the new tool as well in 2.0. The deprecated install_demo_configurations.sh can be completely removed from 3.0.

Describe alternatives you've considered

The alternative approach considered was to keep the name and the input arguments for the installation tool (install_demo_configurations.sh) same to keep the user experience similar. This would have helped in releasing this as a minor version update since the users would be expected to use the tool in the same way as earlier.

However, this still changes some underlying assumptions as it deprecates previously hardcoded demo certificates and this could come as a surprise to the user for a minor version update.

Additional context

No response

[setiah]

A few open questions

1. Should we mark the existing tool as deprecated in next minor release 1.x or in next major release 2.0?
2. Do we see any reason to continue supporting demo certs?

for 1, if we chose to keep it as deprecated in 2.0, we would want to implement the localhost binding change https://github.com/opensearch-project/security/issues/1626 for security sake.

[peterzhuamazon]

[Triage] I have seen https://github.com/opensearch-project/security/pull/1530 merged then get reverted in https://github.com/opensearch-project/security/pull/1637.

Seems like there are still a lot of things to do with little information.

Please advice on the next steps and give more information on how to properly roll out this change. Thanks.

[setiah]

@peterzhuamazon thanks for looking, answering your concerns one at a time..

[Triage] I have seen opensearch-project/security#1530 merged then get reverted in opensearch-project/security#1637.

I am seeking more information on this https://github.com/opensearch-project/security/pull/1530#issuecomment-1035694372

Seems like there are still a lot of things to do with little information.

Sure, happy to answer if you have any specific questions. The intent here is to release the new setup experience for OpenSearch. The issue describes what changes are going in (PRs/Issue(s) attached) and what can be deemed as breaking vs non-breaking. Based on that, it tries to rationalize why these changes should be part of major version release 2.0. Also there are a few open questions, that'll help define finer details better. Are there any concrete points or a release question format that you'd like me to cover? I am not aware if engineering effectiveness has a pre-defined question set for release candidates, so I came up w/ something considering relevant details.

Please advice on the next steps and give more information on how to properly roll out this change.

The next step is to get an Ack on this high level release plan. We can get into any finer details once we start having a discussion around the open questions https://github.com/opensearch-project/opensearch-build/issues/1649#issuecomment-1047386320.

[setiah]

Does the proposal sound like a reasonable plan for the new out of box experience release?

@CEHENKLE @stockholmux @jimishs @davidlago

[setiah]

Having a good out of box experience for security is the first step towards making security a first class citizen in OpenSearch. We should get this out in 2.0, since we also want to make security part of the core soon.

[stockholmux]

@setiah I'm fine with the overall approach. I think the journey to proposed end result is my current concern.

The SemVer spec has a few points of guidance on deprecating functionality. I think I'd need to see a little more detail in your proposal based on that:

1) You need to show how you are going to update the documentation for this deprecation. 'Documentation' probably also should extend to markdown files in the various repos (including the website). Effectively, any place we are saying "use admin:admin" or any other setup assumptions should be noted that this will be deprecated and changed in 2.0.0. I think the impact of this needs to be identified in your proposal.

2) The enforcement of "strong" passwords vs the default credentials causes me concern about the impact this will have on the time-to-first interaction with OpenSearch in a demo or trial scenario. I can dig up data (and have witnessed first hand) that literal first-touch users struggle a lot with authentication even with admin:admin. There are lots of document situations where users end up removing auth entirely as it causes too much friction. I'm worried that, as written, this proposal might increase the friction for an artificial security gain (it's very questionable to need strong passwords on an instance that can't connect to the outside world). Instead of enforcing 'strong' passwords bluntly, can you address the problem (production instances with admin:admin) more precisely?

[setiah]

Hey @stockholmux, thanks for the feedback. There are a few things I would like to clarify that should hopefully make it easier for us to decide.

The SemVer spec has a few points of guidance on deprecating functionality

Good perspective. SemVer provides guidelines for deprecating public APIs or any feature. We are not changing any public APIs and neither are we deprecating any functionality in OpenSearch. We are only changing user experience around the setup process for OpenSearch, which includes - a) setting up certificates, b) setting up passwords. RN, this is done automatically (but insecurely with no guardrails to prevent production use) with install_demo_configurations.sh tool. We would change this with a new tool and some code-changes in security plugin. To me, the problem is more about communicating the new setup experience to users.

(it's very questionable to need strong passwords on an instance that can't connect to the outside world)

The problem lies here. The instance with default admin:admin password "can" connect to the outside world and can be used as a production instance with weak defaults. Also, OpenSearch cannot scan and identify admin:admin password configuration in a running system due to the way passwords are stored (as hashes). I agree it does add an extra setup for users to setup passwords but we are trying to make this step super easy with the tool that also auto-generates strong passwords for users. Like you said, the devil lies in the details, so lets chat again once @ryanbogan puts up the new UX with the password tool.

You need to show how you are going to update the documentation for this deprecation.

The documentation would follow the same release plan since (I believe) OpenSearch documentation is for every version. The version in which we mark the existing tool for deprecation, the document would highlight the same, and announce the new mechanism coming in the next version and how to use it. In the next version where the tool is deprecated, the documentation would just cover the new way of installing security settings.

'Documentation' probably also should extend to markdown files in the various repos (including the website). Effectively, any place we are saying "use admin:admin" or any other setup assumptions should be noted that this will be deprecated and changed in 2.0.0.

Agree

[setiah]

Updated Proposal

(highlighting more details around how this impacts existing and new users and their use cases)

Note

1. New node addition - does not require to run install_demo_configurations.sh tool. It requires importing settings from existing nodes.
2. Version upgrade - does not require to run install_demo_configurations.sh tool. It requires importing settings from existing nodes.

Re: demo certificates and the install_demo_configurations.sh tool

Proposal is to provide two modes of operation for install_demo_configurations.sh tool. In the new mode, it basically generates and configures the new self-signed certificates, while in old mode, it installs the demo certificates (as the existing behavior).

This does not impact new node addition and version upgrade use cases. The tool is required to be run only for first time OpenSearch installation. It is not required when adding a new node or upgrading to a new version because in both these cases, the config settings need to be imported from existing cluster nodes and should not be setup via the tool (though it works for some cases).

Moving forward, the plan is to eventually deprecate this tool in 3.0 in favor of a more unified tool that is easier to discover and extend. Such a tool would help admin with initial cluster setup, adding more nodes, version upgrade, etc. It could be extended via plugins/extensions for their respective post install setup use cases.

Re: removing admin:admin default credentials

Default passwords are configured yml file. The proposal is to remove the default admin:admin password in internal_users.yml file. In addition, we’ll add warnings/deprecation messages in logs/Dashboards for weak configuration like admin:admin, to eventually deprecate those in 3.0.

This does not impact new node addition, or version upgrade from 1.x to 2.0 in any way.

For new users (who start with 2.0), it would help them get started with better defaults. The install_demo_configurations.sh tool can also cater to setting strong passwords (it does not today). Additionally a new password setup tool will be available to also change/setup passwords after the cluster is up.

The reason for adding only warning messages at this point is to provide an easier upgrade path from 1.x to 2.0 and not break the behavior for users who are using default configurations for whatever reasons. The idea is to lean on the side of caution and eventually deprecate the admin:admin password in 3.0.

One alternative is to continue providing admin:admin password as default in yml but add deprecation warnings. However, this does not mitigate the risks as new users would continue to use weak defaults.

Another alternative to put guardrails to prevent production use of admin:admin, however, this breaks the version upgrade case from 1.x to 2.0, so considering this out.

[setiah]

Couple of feedback points from a discussion w/ @dblock @peternied

1. For https://github.com/opensearch-project/OpenSearch/issues/1633, Instead of using --old-mode as an option, which is context dependent, we should use more meaningful context-independent option like --mode=use-demo-certs or --mode=use-self-signed-certs
2. Both PRs related to certificate changes can go in a minor version release if we keep the defaults same i.e. still generate demo certificates, but providing a new mode to generate secure self-signed certs. This can be supplemented with educating users to not use insecure configurations (when they setup) and recommending them to switch to running the tool in --mode=use-self-signed-certs. We can later decide if we want to backport this to 2.0
3. For password setup, instead of releasing a separate shell tool, we could start consolidating the new functions under a new unified tool opensearch-admin that can be extended to include other security setup functions as well. This is already in the long term plan, and we could prioritize this now over later.

[dblock]

I recommend using nouns and a more meaningful name than mode, e.g. --ssl-certs=demo or -ssl-certs=self-signed.

[peterzhuamazon]

[Triage] Hi @dblock what do you think about this issue for now? Is there any more steps we can do to improve given problem statement? Or is this issue outdated for now. Thanks.

[dblock]

AFAIK we only did the admin:admin password part, did we do anything about the rest (certs)?

[smortex]

AFAIK we only did the admin:admin password part, did we do anything about the rest (certs)?

@peterzhuamazon, @dblock I was speaking about this on slack yesterday and was asked to open a RFC about this. It can be found here: https://github.com/opensearch-project/security/issues/4344