search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.52k stars 1.75k forks source link

[BUG] Opensearch 2.9.0 - Failed to initialize LogType config index and builtin log types #9061

Open crazymind1337 opened 1 year ago

crazymind1337 commented 1 year ago

Describe the bug I am trying to install the new version of opensearch on Ubuntu 20.04. But the cluster state stays at yellow.

To Reproduce Steps to reproduce the behavior:

  1. Install opensearch 2.9.0 on Ubuntu 20.04
  2. configure a basic setting: 
    cluster.name: cluster
http.port: 9200
network.host: <local-ip>
node.name: test-node-1
path.data: "/var/lib/opensearch"
path.logs: "/var/log/opensearch"
plugins.security.disabled: true
transport.host: <local-ip>
transport.tcp.port: 9300
cluster.initial_cluster_manager_nodes: test-node-1
discovery.seed_hosts: test-node-1
  3. start opensearch service
  4. check logs

Expected behavior Cluster state becomes green.

 ~# curl --silent --connect-timeout 3 --header 'Host: stats' --write-out '%{http_code}' "http://<local-ip>:9200/_cluster/health?pretty=true"
{
  "cluster_name" : "cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 57.14285714285714
}

Plugins Basic installation.

Screenshots %

Host/Environment (please complete the following information):

Additional context In the logs of opensearch I am able to discover the following informations:

[2023-08-02T11:07:51,126][ERROR][o.o.s.u.SecurityAnalyticsException] [test-node-1] Security Analytics error:
org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-pre-packaged-rules-config/CWaJHMTdTlmZSg9zl5rPaA] already exists
[2023-08-02T11:07:51,149][WARN ][o.o.s.SecurityAnalyticsPlugin] [test-node-1] Failed initializing prepackaged rules
org.opensearch.securityanalytics.util.SecurityAnalyticsException: index [.opensearch-sap-pre-packaged-rules-config/CWaJHMTdTlmZSg9zl5rPaA] already exists
[2023-08-02T11:07:51,196][WARN ][o.o.s.SecurityAnalyticsPlugin] [test-node-1] Failed to initialize LogType config index and builtin log types
[2023-08-02T11:07:52,152][INFO ][o.o.c.r.a.AllocationService] [test-node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]).

Complete log can be found at: https://justpaste.it/c9c32

dbwiddis commented 1 year ago 
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "unassigned_shards" : 3,

From the docs:

A yellow cluster status means the primary shards for all indexes are allocated to nodes in a cluster, but the replica shards for at least one index aren't.

You only have one node, but the replica shard needs to be on a different node.

rauldpm commented 11 months ago

I have replicated this behavior with the RPM package (fresh install), this does not happen in 2.8.0, I also get the same log errors/warnings

OpenSearch 2.9.0 fresh install, cluster status and shards ``` [root@centos7 vagrant]# yum install opensearch-2.9.0-linux-x64.rpm -y Loaded plugins: fastestmirror Examining opensearch-2.9.0-linux-x64.rpm: opensearch-2.9.0-1.x86_64 Marking opensearch-2.9.0-linux-x64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package opensearch.x86_64 0:2.9.0-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================================================================= Installing: opensearch x86_64 2.9.0-1 /opensearch-2.9.0-linux-x64 993 M Transaction Summary ============================================================================================================================================================================================================================================= Install 1 Package Total size: 993 M Installed size: 993 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : opensearch-2.9.0-1.x86_64 1/1 ### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable opensearch.service ### You can start opensearch service by executing sudo systemctl start opensearch.service ### Create opensearch demo certificates in /etc/opensearch/ See demo certs creation log in /var/log/opensearch/install_demo_configuration.log Verifying : opensearch-2.9.0-1.x86_64 1/1 Installed: opensearch.x86_64 0:2.9.0-1 Complete! [root@centos7 vagrant]# sudo systemctl start opensearch [root@centos7 vagrant]# sudo systemctl status opensearch ● opensearch.service - OpenSearch Loaded: loaded (/usr/lib/systemd/system/opensearch.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2023-10-11 15:33:17 UTC; 15s ago Docs: https://opensearch.org/ Main PID: 5633 (java) CGroup: /system.slice/opensearch.service └─5633 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Dj... Oct 11 15:33:07 centos7 systemd[1]: Starting OpenSearch... Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: A terminally deprecated method in java.lang.System has been called Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.9.0.jar) Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager will be removed in a future release Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: A terminally deprecated method in java.lang.System has been called Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.9.0.jar) Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager will be removed in a future release Oct 11 15:33:17 centos7 systemd[1]: Started OpenSearch. ``` ``` [root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cluster/health?pretty { "cluster_name" : "opensearch", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 5, "active_shards" : 5, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 3, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 62.5 } [root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/shards .opensearch-observability 0 p STARTED 0 208b 127.0.0.1 centos7 .plugins-ml-config 0 p STARTED 1 3.8kb 127.0.0.1 centos7 .plugins-ml-config 0 r UNASSIGNED .opensearch-sap-pre-packaged-rules-config 0 p STARTED 127.0.0.1 centos7 .opensearch-sap-pre-packaged-rules-config 0 r UNASSIGNED .opensearch-sap-log-types-config 0 p STARTED 127.0.0.1 centos7 .opensearch-sap-log-types-config 0 r UNASSIGNED .opendistro_security 0 p STARTED 10 74.8kb 127.0.0.1 centos7 ```

Apparently, the unassigned shards are duplicated:

.plugins-ml-config                        0 r UNASSIGNED           
.opensearch-sap-pre-packaged-rules-config 0 r UNASSIGNED            
.opensearch-sap-log-types-config          0 r UNASSIGNED

Note: This appears to be solved in 2.10.0

OpenSearch 2.10.0 fresh install ``` [root@centos7 vagrant]# yum localinstall opensearch-2.10.0-linux-x64.rpm -y Loaded plugins: fastestmirror Examining opensearch-2.10.0-linux-x64.rpm: opensearch-2.10.0-1.x86_64 Marking opensearch-2.10.0-linux-x64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package opensearch.x86_64 0:2.10.0-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================================================================= Installing: opensearch x86_64 2.10.0-1 /opensearch-2.10.0-linux-x64 1.0 G Transaction Summary ============================================================================================================================================================================================================================================= Install 1 Package Total size: 1.0 G Installed size: 1.0 G Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : opensearch-2.10.0-1.x86_64 1/1 ### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable opensearch.service ### You can start opensearch service by executing sudo systemctl start opensearch.service ### Create opensearch demo certificates in /etc/opensearch/ See demo certs creation log in /var/log/opensearch/install_demo_configuration.log ### Upcoming breaking change in packaging In a future release of OpenSearch, we plan to change the permissions associated with access to installed files If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043 Verifying : opensearch-2.10.0-1.x86_64 1/1 Installed: opensearch.x86_64 0:2.10.0-1 Complete! [root@centos7 vagrant]# sudo systemctl start opensearch [root@centos7 vagrant]# systemctl status opensearch ● opensearch.service - OpenSearch Loaded: loaded (/usr/lib/systemd/system/opensearch.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2023-10-11 15:51:34 UTC; 3s ago Docs: https://opensearch.org/ Main PID: 24907 (java) CGroup: /system.slice/opensearch.service └─24907 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -D... Oct 11 15:51:25 centos7 systemd[1]: Starting OpenSearch... Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: A terminally deprecated method in java.lang.System has been called Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar) Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager will be removed in a future release Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: A terminally deprecated method in java.lang.System has been called Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar) Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager will be removed in a future release Oct 11 15:51:34 centos7 systemd[1]: Started OpenSearch. ``` ``` [root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cluster/health?pretty { "cluster_name" : "opensearch", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 4, "active_shards" : 4, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } [root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/shards .opensearch-observability 0 p STARTED 0 208b 127.0.0.1 centos7 .opensearch-sap-log-types-config 0 p STARTED 127.0.0.1 centos7 .opendistro_security 0 p STARTED 10 75.4kb 127.0.0.1 centos7 ``` 
[root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/shards
.opensearch-observability        0 p STARTED  0   208b 127.0.0.1 centos7
.opensearch-sap-log-types-config 0 p STARTED           127.0.0.1 centos7
.opendistro_security             0 p STARTED 10 75.4kb 127.0.0.1 centos7
techsk8 commented 11 months ago

indeed, the problem is solved on opensearch 2.10 and 2.11

HoffmannTom commented 10 months ago

The warning "Failed to initialize LogType config index and builtin log types" as mentioned in the issue title is still present in version 2.11 It seems that the creation of the index is always triggered and produces the warning, if index already exists. The cluster status is green though.