[BUG][OpenSearch Dashboards] Use of plugins.installList fails because access denied to plugin images

[gsmith-sas]

Describe the bug I attempted to limit which plugins were loaded with OpenSearch Dashboards by using the plugins.installList key. The OpenSearch Dashboards pod failed to come up and displayed messages indicating the URL (pointing to the plugin zip file) was not valid. When I attempt to access that URL in my browser, it indicates that I do not have access. (Specific error messages included below.)

To Reproduce Steps to reproduce the behavior:

1. I added the following to my user-values.yaml file:

   plugins:
   enabled: true
   installList:
   - opensearch-alerting
   - opensearch-index-management
   - opensearch-job-scheduler
   - opensearch-reports-scheduler
   - opensearch-security
   - opensearch-sql

2. I then deployed OpenSearch Dashboards using the Helm chart and pointing to my user-values.yaml file

3. I confirmed that the values were picked up and used to construct a command within the OpenSearch Dashboard pod defintion (as shown below):

     containers:
       - name: dashboards
         image: opensearchproject/opensearch-dashboards:2.4.1
         command:
           - sh
           - '-c'
           - >
             #!/usr/bin/bash
             set -e
             ./bin/opensearch-dashboards-plugin install opensearch-alerting
             ./bin/opensearch-dashboards-plugin install opensearch-index-management
             ./bin/opensearch-dashboards-plugin install opensearch-job-scheduler
             ./bin/opensearch-dashboards-plugin install opensearch-reports-scheduler
             ./bin/opensearch-dashboards-plugin install opensearch-security
             ./bin/opensearch-dashboards-plugin install opensearch-sql
             bash opensearch-dashboards-docker-entrypoint.sh     opensearch-dashboards

4. I then watched the OpenSearch Dashboards pod logs signs of success. However, I only see the following messages:

   Attempting to transfer from opensearch-alerting
   Attempting to transfer from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/linux/x64/tar/builds/opensearch-dashboards/plugins/opensearch-alerting-2.4.1.zip
   Plugin installation was unsuccessful due to error "No valid url specified."

5. When I attempt to access that URL via my browser, it fails with the following XML error indicating that I do not have access

   <Error>
   <Code>AccessDenied</Code>
   <Message>Access Denied</Message>
   <RequestId>RCBX79B9PSDAPH2H</RequestId>
   <HostId>
   UEXoSll4fjiMSQOUNb3y59M94H9p8U/4U0wvoptBP4UMUpwrKa+gPQ5NkWRTA/fVMCUvEUAsnhE=
   </HostId>
   </Error>

Expected behavior I had expected that OpenSearch Dashboards would deploy successfully and only include the plugins I had specified

Chart Name OpenSearch Dashboards (version 2.7.0)

Host/Environment (please complete the following information):

• Helm Version: 3.7.0
• Kubernetes Version: 1.22.3

Additional context Add any other context about the problem here.

[rishabh6788]

Will debug this.

[rishabh6788]

@gsmith-sas It seems you are trying to install OpenSearch plugins inside OpenSearch Dashboards docker image. Please use the OpenSearch chart to install opensearch supported plugins.

[prudhvigodithi]

Hey @gsmith-sas FYI the dashboard plugins list https://opensearch.org/docs/latest/install-and-configure/install-dashboards/plugins#available-plugins. Thank you

[zelinh]

Hey @gsmith-sas you could try with for example ./bin/opensearch-dashboards-plugin install alertingDashboards to install dashboards plugin. https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/linux/x64/tar/builds/opensearch-dashboards/plugins/alertingDashboards-2.4.1.zip is an available URL link.

[gsmith-sas]

@rishabh6788 @prudhvigodithi @zelinh Thank you all for responding and pointing out I was trying to use OpenSearch plugins with OpenSearch Dashboards. I just tried again being careful to only list specific OpenSearch Dashboards plugins in the user-values.yaml file passed to the OpenSearch Dashboards helm chart. I got a little further along this time but still no success. The OpenSearch Dashboards pod attempted to start and, looking at the pod logs, I can see that it was able to download the Alerting dashboards plugin. Here are the pod log messages:

Attempting to transfer from alertingDashboards
Attempting to transfer from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/linux/x64/tar/builds/opensearch-dashboards/plugins/alertingDashboards-2.4.1.zip
Transferring 7147208 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin alertingDashboards already exists, please remove before installing a new version

The OpenSearch Dashboards pod also restarts after this. Most of the time, the messages about transferring and extracting the plugin archive appear twice. But it isn't clear to me whether that happens twice before the pod restarts or if the 2nd set of messages is from after the restart.

By the way, I am using the publicly available OpenSearch Dashboards container image (opensearchproject/opensearch-dashboards:2.4.1). Does that image have all of the plugins already installed? If so, is there an alternative container image that does not have plugins pre-installed?

UPDATE: I just noticed this comment in the charts/opensearch-dashboards/values.yaml file: ## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearchDashboards image. Does this mean that these Helm chart keys can NOT be used to disable any of the "standard" OpenSearch Dashboards plugins (e.g. Anomaly Detection, Observability, etc.)? I was really hoping I could disable some of that since my application doesn't make use of those capabilities and the options will just confuse my users.

[rishabh6788]

@gsmith-sas The OpenSearch and OpensSearch Dashbords images come pre-installed with all the plugins. Not sure why you are trying to install again?

[gsmith-sas]

@rishabh6788 I wasn't (intentionally) trying to re-install all of the plugins. But I was hoping to limit which ones were loaded. As I mentioned in my updated comments, our application doesn't make use of several plugins (e.g. Anomaly Detection, Observability, etc.). I was hoping to exclude those plugins from the installation to eliminate menu items related to those plugins from appearing in OpenSearch Dashboards. That would help us minimize confusion among our end-users and eliminate the need to tell them to ignore various options.

Is is possible to exclude certain plugins when deploying via the Helm charts? Or, alternatively, is is possible to use security settings to "hide" options that the user doesn't have access to?

[rishabh6788]

@gsmith-sas to remove plugins you can just replace install with remove keyword in your command, e.g., ./bin/opensearch-dashboards-plugin remove <plugin-name>. Let me know if that helps.

[gsmith-sas]

Thanks @rishabh6788. But I'm trying to use what the Helm chart supports and it doesn't seem to support a plugins.excludelist option. Maybe that's something I should open a feature request for. But would that work? If the pod's container definition already includes all of the 'standard' plugins, could we still remove them using the command you shared? If so, I suspect it could be implemented similar to the existing logic for deploying additional plugins.

[rishabh6788]

@prudhvigodithi Can you please take a look at above request?

[prudhvigodithi]

Hey @gsmith-sas thanks for the explanation, the plugins cannot be removed using helm chart, the helm chart purpose is to install new plugins from the given defaults, also since the OpenSearch dashboard bundle comes with the default, the official helm chart of the OpenSearch dashboard should honor them.

Secondly removing the defaults is not recommended, also adding a flag to add/remove in helm chart is dangerous switch, as it could remove the installed settings when chart is redeployed.

With this we have 2 options. 1) Build a custom Image of your choice of installed plugins (by removing the not required ones). 2) Hack the postStart from the chart lifecycle settings to remove the plugin.

Tagging other maintainers to get some more feedback. @bbarani @peterzhuamazon @TheAlgo @DandyDeveloper