search

wazuh / wazuh-kubernetes

Wazuh - Wazuh Kubernetes
https://wazuh.com/
GNU General Public License v2.0
268 stars 164 forks source link

Issues after upgrade from 4.2 to 4.3 #299

Open andraspavelbaystream opened 2 years ago

andraspavelbaystream commented 2 years ago

Hello,

I did the upgrade from 4.2 to 4.3 based on the following guide: https://github.com/wazuh/wazuh-kubernetes/blob/migration-doc/upgrade-odfe-indexer.md I have two issues remained.

Issue 1: I have since the upgrade the following error repeteadly in the wazuh indexer:

[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"IgSsu0kWT9yBKU7ePOmC-Q#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[IgSsu0kWT9yBKU7ePOmC-Q#metadata]: version conflict, document already exists (current version [28501])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[IgSsu0kWT9yBKU7ePOmC-Q#metadata]: version conflict, document already exists (current version [28501])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"RKofUTRdSTa_gp0oAW2bSA#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[RKofUTRdSTa_gp0oAW2bSA#metadata]: version conflict, document already exists (current version [28480])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[RKofUTRdSTa_gp0oAW2bSA#metadata]: version conflict, document already exists (current version [28480])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"U36GnbehTBuswingPx7dsQ#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[U36GnbehTBuswingPx7dsQ#metadata]: version conflict, document already exists (current version [28501])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[U36GnbehTBuswingPx7dsQ#metadata]: version conflict, document already exists (current version [28501])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"yZlVmcDqQH-i9mzWTmhytA#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[yZlVmcDqQH-i9mzWTmhytA#metadata]: version conflict, document already exists (current version [28481])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[yZlVmcDqQH-i9mzWTmhytA#metadata]: version conflict, document already exists (current version [28481])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"QhMS7ZXsQjeBhz-3H7MiLA#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[QhMS7ZXsQjeBhz-3H7MiLA#metadata]: version conflict, document already exists (current version [28503])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[QhMS7ZXsQjeBhz-3H7MiLA#metadata]: version conflict, document already exists (current version [28503])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"yvSIN3nlRxu47eFlfxYHkA#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[yvSIN3nlRxu47eFlfxYHkA#metadata]: version conflict, document already exists (current version [28501])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[yvSIN3nlRxu47eFlfxYHkA#metadata]: version conflict, document already exists (current version [28501])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"nWW1DJpHTe-epP7XJ-ZOMA#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[nWW1DJpHTe-epP7XJ-ZOMA#metadata]: version conflict, document already exists (current version [28503])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[nWW1DJpHTe-epP7XJ-ZOMA#metadata]: version conflict, document already exists (current version [28503])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"rOuEZbhfTgWsjGe4T-yvTQ#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[rOuEZbhfTgWsjGe4T-yvTQ#metadata]: version conflict, document already exists (current version [28501])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[rOuEZbhfTgWsjGe4T-yvTQ#metadata]: version conflict, document already exists (current version [28501])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"ybuae34bR5GXjEaD5rkfMA#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[ybuae34bR5GXjEaD5rkfMA#metadata]: version conflict, document already exists (current version [28480])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[ybuae34bR5GXjEaD5rkfMA#metadata]: version conflict, document already exists (current version [28480])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"XtjBnVnfQqikZS-Dc3VAlQ#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[XtjBnVnfQqikZS-Dc3VAlQ#metadata]: version conflict, document already exists (current version [28503])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[XtjBnVnfQqikZS-Dc3VAlQ#metadata]: version conflict, document already exists (current version [28503])]
[2022-09-07T14:52:39,684][ERROR][o.o.i.i.MetadataService  ] [wazuh-indexer-0] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"nowqMLAKQtCQGI4OkaDupg#metadata","cause":{"type":"version_conflict_engine_exception","reason":"[nowqMLAKQtCQGI4OkaDupg#metadata]: version conflict, document already exists (current version [28503])","index":".opendistro-ism-config","shard":"0","index_uuid":"BiNAraAFTf66bZKaIjC_yA"},"status":409}, [.opendistro-ism-config/BiNAraAFTf66bZKaIjC_yA][[.opendistro-ism-config][0]] VersionConflictEngineException[[nowqMLAKQtCQGI4OkaDupg#metadata]: version conflict, document already exists (current version [28503])]

Issue 2:

I have these errors in the /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log log file repeteadly:

Sep 7, 2022 @ 16:15:00  ERROR  Could not check if the index wazuh-monitoring-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:15:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:15:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:20:01  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:20:01  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:25:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:25:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:30:00  ERROR  Could not check if the index wazuh-monitoring-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:30:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:30:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:35:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:35:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:40:01  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:40:01  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:45:00  ERROR  Could not check if the index wazuh-monitoring-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:45:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:45:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:50:01  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:50:01  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:55:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check
Sep 7, 2022 @ 16:55:00  INFO  Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check

I'd like to solve these two issues, these are the only remained problems in our environment.

Could somebody help me with this please?

Thanks, Andras

vcerenu commented 2 years ago

Hello @andraspavelbaystream

Do you have the data of the PVCs that you created before starting the migration and the current data of the new PVCs that you created with the previous volume?

Due to the errors that are being logged, it seems that the owner of the service does not have permissions on the previous indexes, so I ask you if you can enter the pod and verify the permissions of the index files.

andraspavelbaystream commented 2 years ago

Hello @vcerenu

Thanks for your reply.

I see the old data before the migration, and the new data is saved well.

From the logs it looks like it is affecting only the wazuh-statistics- and wazuh-monitoring- indicies.

I had the same idea (file permission issue), but for the new, and exisiting data files the permissions are 664 (rw+rw+r), and the user and group is also the "wazuh-indexer". See below.

Where else I can check for permission problems?

Thank you!

image

vcerenu commented 2 years ago

Hello @andraspavelbaystream

About the error message:

Sep 7, 2022 @ 16:55:00 INFO Could not check if the index wazuh-statistics-2022.36w exists due to no permissions for create, delete or check

You should check the permissions of the role assigned to the user you use to connect to the Wazuh API (the default user is wazuh_wui) on the index wazuh-statistics-2022.36w. In case you do not have permissions on the index, you should add it.

Regarding the other error, I'm still analyzing.

andraspavelbaystream commented 2 years ago

Hi @vcerenu

Thank you!

I was looking the permissions, spin up a new test environment via version 4.3.7, and checked the indexer permissions. The wazuh_ui_admin and the wazuh_ui_user roles were missing. (see below the entries) I created them using the same config as the test environment, and that did the trick, now the necessary indices are created successfully. I guess this should be added to the migration doc :)

image

Regarding the other issue with .opendistro-ism-config, no changes, that is still happening.

Thanks!

andraspavelbaystream commented 2 years ago

Hi @vcerenu

Do you have any news on the .opendistro-ism-config issue?

Thank you! Andras