[Bug] Elasticsearch experiences Bulk write of deprecation logs encountered some failures

[tschaffter]

Is there an existing issue for this?

• [X] I have searched the existing issues

What product(s) are you seeing the problem on?

OpenChallenges

Current behavior

1. openchallenges-build-images
2. nx serve-detach openchallenges-organization-service

The three ES instances fail to start. The 2nd and 3rd nodes fail because the first node fails. The logs of the first node end with:

{"type": "server", "timestamp": "2023-09-05T01:10:35,197Z", "level": "ERROR", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "openchallenges-elasticsearch", "node.name": "openchallenges-elasticsearch", "message": "Bulk write of deprecation logs encountered some failures: [[aAnjYooBlfoVQw1-QYdS UnavailableShardsException[[.ds-.logs-deprecation.elasticsearch-default-2023.09.05-000001][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-.logs-deprecation.elasticsearch-default-2023.09.05-000001][0]] containing [index {[.logs-deprecation.elasticsearch-default][_doc][aAnjYooBlfoVQw1-QYdS], source[{\"event.dataset\": \"deprecation.elasticsearch\", \"@timestamp\": \"2023-09-05T01:09:33,754Z\", \"log.level\": \"CRITICAL\", \"log.logger\": \"org.elasticsearch.deprecation.xpack.security.support.SecurityStatusChangeListener\", \"elasticsearch.cluster.name\": \"openchallenges-elasticsearch\", \"elasticsearch.cluster.uuid\": \"\", \"elasticsearch.node.id\": \"\", \"elasticsearch.node.name\": \"openchallenges-elasticsearch\", \"trace.id\": \"\", \"message\": \"The default behavior of disabling security on basic licenses is deprecated. In a later version of Elasticsearch, the value of [xpack.security.enabled] will default to \\\"true\\\" , regardless of the license level. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security, or explicitly disable security by setting [xpack.security.enabled] to false in elasticsearch.yml\", \"data_stream.type\": \"logs\", \"data_stream.dataset\": \"deprecation.elasticsearch\", \"data_stream.namespace\": \"default\", \"ecs.version\": \"1.7\", \"elasticsearch.event.category\": \"security\", \"event.code\": \"security_implicitly_disabled\", \"elasticsearch.http.request.x_opaque_id\": \"\" }\n]}]]]]]", "cluster.uuid": "wZ2xG9a6SgaKDagsEcmNQg", "node.id": "ZtDSIXS0SUik_RGVJUdPWA"  }

Expected behavior

No response

Anything else?

Experienced when working on #2079

Commit ID

No response

Are you developing inside the dev container?

• [X] I am using the dev container

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[tschaffter]

There is almost no space left on the EC2 instance:

vscode@7047b27f8ebb:/workspaces/sage-monorepo$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          80G   74G  6.1G  93% /

Removing all Docker artifacts in the dev container does not free as much as expected:

vscode@7047b27f8ebb:/workspaces/sage-monorepo$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          80G   58G   23G  72% /

Removing all the images on the host saves 23G:

[ec2-user@ip-10-41-30-105 sage-monorepo]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  596K   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/nvme0n1p1   80G   37G   44G  46% /
tmpfs           3.2G     0  3.2G   0% /run/user/1000

[tschaffter]

Conclusion

After making some space, the OC stack started successfully, including the Elasticsearch nodes.