Unable to open the Opensearch Dashboards

[Arkadij-RS]

Hello,

I am trying to add Opensearch container to my DDEV environment serving Magento 2.4.7. The DDEV works in Ubuntu 20.04 environment and version is:

ITEM VALUE DDEV version v1.23.5 architecture amd64 cgo_enabled 0 db ddev/ddev-dbserver-mariadb-10.11:v1.23.5 ddev-ssh-agent ddev/ddev-ssh-agent:v1.23.5 docker 27.3.1 docker-api 1.47 docker-compose v2.29.7 docker-platform linux-docker global-ddev-dir /home/arkadij/.ddev mutagen 0.17.2 os linux router ddev/ddev-traefik-router:v1.23.5 web ddev/ddev-webserver:v1.23.5

The Opensearch itself returns

{ "name" : "opensearch-node", "cluster_name" : "opensearch-cluster", "cluster_uuid" : "GAeIrkTSQ1OR_3ArIJpQ2A", "version" : { "distribution" : "opensearch", "number" : "2.18.0", "build_type" : "tar", "build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547", "build_date" : "2024-10-31T19:08:39.157471098Z", "build_snapshot" : false, "lucene_version" : "9.12.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" }

But the Dashboard does not work:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

In the logs I can see this error:

{"type":"log","@timestamp":"2024-11-20T22:05:27Z","tags":["error","opensearch","data"],"pid":1,"message":"[cluster_block_exception]: index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"} {"type":"log","@timestamp":"2024-11-20T22:05:27Z","tags":["error","http"],"pid":1,"message":"ResponseError: index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];: cluster_block_exception: [cluster_block_exception] Reason: index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];\n at onBody (/usr/share/opensearch-dashboards/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n at IncomingMessage.onEnd (/usr/share/opensearch-dashboards/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n at IncomingMessage.emit (node:events:529:35)\n at IncomingMessage.emit (node:domain:489:12)\n at endReadableNT (node:internal/streams/readable:1400:12)\n at processTicksAndRejections (node:internal/process/task_queues:82:21) {\n meta: {\n body: { error: [Object], status: 429 },\n statusCode: 429,\n headers: {\n 'x-opaque-id': '39e26fa2-b58c-41d5-a267-eedec3a22428',\n 'content-type': 'application/json; charset=UTF-8',\n 'content-length': '401'\n },\n meta: {\n context: null,\n request: [Object],\n name: 'opensearch-js',\n connection: [Object],\n attempts: 0,\n aborted: false\n }\n },\n isBoom: true,\n isServer: false,\n data: null,\n output: {\n statusCode: 429,\n payload: {\n message: 'index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];: cluster_block_exception: [cluster_block_exception] Reason: index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];',\n statusCode: 429,\n error: 'Too Many Requests'\n },\n headers: {}\n },\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/tooManyRequests'\n}"} {"type":"error","@timestamp":"2024-11-20T22:05:27Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:186:34)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://project-local.ddev.site:5602/","message":"Internal Server Error"}

[cmuench]

@Arkadij-RS This can happen if there is less disk space left on a device. Can you verify that?

[cmuench]

Then try to remove the read-only block.

curl -XPUT 'http://<opensearch-host>:9200/_all/_settings' -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'

[Arkadij-RS]

Thank you for your quick response Christian,

Could be the disk space. I have only 8% free on my SSD which means 36GB. For this particular project should pretty enough.

However, after removing the read-only block mu Dashboard was able to open. But some index-rebuild operations return the same

TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

Is it possible to set Opensearch check the absolute amount of disk space available and not the percentage?

[cmuench]

@Arkadij-RS Yes, it's possible by changing the config.

Have a look here: https://opster.com/guides/elasticsearch/capacity-planning/elasticsearch-low-disk-watermark/

For opensearch it's the opensearch.yml.