search

wazuh / wazuh-indexer

Wazuh indexer, the Wazuh search engine
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
6 stars 16 forks source link

Wazuh Indexer failed to start: SSL Cert exception error #148

Closed briantownjr closed 5 months ago

briantownjr commented 5 months ago

Describe the bug Wazuh-indexer service will not start

To Reproduce Updated to 4.7.2.

Expected behavior A clear and concise description of what you expected to happen.

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

Additional context Getting below errors when doing a status against the wazuh-indexer service:

[2024-02-05T19:34:36,844][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [1044762216]

[2024-02-05T19:34:36,858][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1044762216], after [1022933632], allocations [1], duration [13]

[2024-02-05T19:34:44,222][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [1042856576]

[2024-02-05T19:34:44,231][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1042856576], after [1022860520], allocations [1], duration [9]

I modified my jvm.options file to be set to 8g, checked it via this command:grep -i "gc,init" /var/log/wazuh-indexer/gc.log

[2024-02-05T20:07:40.647+0000][8338][gc,init] Version: 17.0.7+7 (release)

[2024-02-05T20:07:40.647+0000][8338][gc,init] CPUs: 4 total, 4 available

[2024-02-05T20:07:40.647+0000][8338][gc,init] Memory: 15966M

[2024-02-05T20:07:40.647+0000][8338][gc,init] Large Page Support: Disabled

[2024-02-05T20:07:40.647+0000][8338][gc,init] NUMA Support: Disabled

[2024-02-05T20:07:40.647+0000][8338][gc,init] Compressed Oops: Enabled (Zero based)

[2024-02-05T20:07:40.647+0000][8338][gc,init] Heap Region Size: 4M

[2024-02-05T20:07:40.647+0000][8338][gc,init] Heap Min Capacity: 8G

[2024-02-05T20:07:40.647+0000][8338][gc,init] Heap Initial Capacity: 8G

[2024-02-05T20:07:40.647+0000][8338][gc,init] Heap Max Capacity: 8G

[2024-02-05T20:07:40.647+0000][8338][gc,init] Pre-touch: Enabled

[2024-02-05T20:07:40.647+0000][8338][gc,init] Parallel Workers: 4

[2024-02-05T20:07:40.647+0000][8338][gc,init] Concurrent Workers: 1

[2024-02-05T20:07:40.647+0000][8338][gc,init] Concurrent Refinement Workers: 4

[2024-02-05T20:07:40.647+0000][8338][gc,init] Periodic GC: Disabled

Also followed guides here: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html

Not sure what the root cause is.

Noticing this issue as well:

Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

Seems might be an issue with JDK update and TLS versions?

Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remainin

g for AEAD cipher fragment (2). Needs to be more than tag size (16)

javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]

at sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]

at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]

at sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]

at sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]

at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]

jvm.options:

JVM configuration

################################################################

IMPORTANT: JVM heap size

################################################################

You should always set the min and max JVM heap

size to the same value. For example, to set

the heap to 4 GB, set:

-Xms4g

-Xmx4g

See https://opensearch.org/docs/opensearch/install/important-settings/

for more information

################################################################

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms4g

-Xmx4g

-Xms8g -Xmx8g

rst-wazuh-cluster.log gc.log

briantownjr commented 5 months ago

Hmm, actually poking around more I noticed this in the logs:

Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup/action_groups.yml

However wazuh-indexer def has permissions on that file:

ls -ltr /etc/wazuh-indexer/backup/

total 68

-rw------- 1 wazuh-indexer wazuh-indexer 153 Feb 5 19:32 whitelist.yml

-rw------- 1 wazuh-indexer wazuh-indexer 125 Feb 5 19:32 tenants.yml

-rw------- 1 wazuh-indexer wazuh-indexer 3617 Feb 5 19:32 roles.yml

-rw------- 1 wazuh-indexer wazuh-indexer 1236 Feb 5 19:32 roles_mapping.yml

-rw------- 1 wazuh-indexer wazuh-indexer 12615 Feb 5 19:32 opensearch.yml.example

-rw------- 1 wazuh-indexer wazuh-indexer 49 Feb 5 19:32 nodes_dn.yml

-rw------- 1 wazuh-indexer wazuh-indexer 1253 Feb 5 19:32 internal_users.yml

-rw------- 1 wazuh-indexer wazuh-indexer 10063 Feb 5 19:32 config.yml.dpkg-dist

-rw------- 1 wazuh-indexer wazuh-indexer 3845 Feb 5 19:32 config.yml

-rw------- 1 wazuh-indexer wazuh-indexer 795 Feb 5 19:32 audit.yml

-rw------- 1 wazuh-indexer wazuh-indexer 1973 Feb 5 19:32 allowlist.yml

-rw------- 1 wazuh-indexer wazuh-indexer 54 Feb 5 19:32 action_groups.yml

I moved the backup dir entirely out and service was able to restart service and run the password update tool to log back in.