jplangford
commented
2 years ago I should also mention that this is a single node cluster. If that was not already obvious.
Closed jplangford closed 2 years ago
jplangford
commented
2 years ago I should also mention that this is a single node cluster. If that was not already obvious.
vcerenu
commented
2 years ago Hello @jplangford
I would need to know if this error is the only one that is generated in order to verify the status of the stack. Could you attach the logs of the 3 containers to see what happens at the beginning?
The images you are using I see that you get them from another repository, are these the same as the official ones pushed to your own repository or do they have any changes?
Regarding the configurations, are you attaching the opensearch.yml and opensearch_dashboards.yml files on the volumes? Or are you just passing the settings through environment variables? If you are using the configuration files, could you attach them?
Also know what certificates you are using, if they are self-signed and created by the script we provide, know if you are using the same .yml that we provide, with the corresponding ip or dns.
jplangford
commented
2 years ago Hi @vcerenu, thank you for your reply.
Please find attached the logs from the containers (logs.tar.gz). The wazuh container log was full of errors like this one:
2022-06-14T09:02:30.731Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc0a23189267cfe23, ext:84823491, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"2c2e9223-07a7-439c-a63d-4d86155275ef","hostname":"wazuh.manager","id":"46b7957e-2fd5-4748-aa0f-360ad613e474","name":"wazuh.manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"wazuh.manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":3941977},"message":"{\"timestamp\":\"2022-06-14T09:01:41.383+0000\",\"rule\":{\"level\":7,\"description\":\"New dpkg (Debian Package) installed.\",\"id\":\"2902\",\"firedtimes\":1,\"mail\":false,\"groups\":[\"syslog\",\"dpkg\",\"config_changed\"],\"pci_dss\":[\"10.6.1\",\"10.2.7\"],\"gpg13\":[\"4.10\"],\"gdpr\":[\"IV_35.7.d\"],\"hipaa\":[\"164.312.b\"],\"nist_800_53\":[\"AU.6\",\"AU.14\"],\"tsc\":[\"CC7.2\",\"CC7.3\",\"CC6.8\",\"CC8.1\"]},\"agent\":{\"id\":\"001\",\"name\":\"dev-wazuh-manager\",\"ip\":\"192.168.1.2\"},\"manager\":{\"name\":\"wazuh.manager\"},\"id\":\"1655197301.6521556\",\"full_log\":\"2022-06-14 09:01:40 status installed google-cloud-ops-agent:amd64 2.16.0~ubuntu20.04\",\"decoder\":{\"name\":\"dpkg-decoder\"},\"data\":{\"dpkg_status\":\"status installed\",\"package\":\"google-cloud-ops-agent\",\"arch\":\"amd64\",\"version\":\"2.16.0~ubuntu20.04\"},\"location\":\"/var/log/dpkg.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::7078029-2064", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc0008336c0), Source:"/var/ossec/logs/alerts/alerts.json", Offset:3942753, Timestamp:time.Time{wall:0xc0a23189267137f5, ext:84051861, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x6c008d, Device:0x810}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [decoder.name] cannot be changed from type [keyword] to [text]"}However I have redacted them as they have details of our network I am prohibited from sharing.
The images I am using, I download from the official repository and push to our own, there are no modifications. They are simply retagged.
I am using configuration files. The certificates I am using are not generated using your container. I had issues with it a few days back which turned out to be a kernel bug. But by the time I realised that was what was happening, I had rewritten the logic to generate the certificates in Ansible. I would prefer to keep the logic outside containers due to complexities of our environment which I won't bore you with.
I have tar'ed up a development system's configuration for your perusal, please find it attached (config.tar.gz).
Thanks again.
jplangford
commented
2 years ago vcerenu
commented
2 years ago Hello
We should check if the Wazuh templates are inserted in indexer:
curl -XGET "https://wazuh.indexer:9200/_cat/templates" -u admin:<admin_password> -k -s
I was checking the configuration files and within the opensearch.yml file you don't have full paths to the location of the certificates. Inside the file that we add to the deployment we use the environment variable $OPENSEARCH_PATH_CONF, which contains the path of the configuration files:
plugins.security.ssl.http.pemcert_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.pem
plugins.security.ssl.http.pemkey_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.key
plugins.security.ssl.http.pemtrustedcas_filepath: ${OPENSEARCH_PATH_CONF}/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.pem
plugins.security.ssl.transport.pemkey_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.key
plugins.security.ssl.transport.pemtrustedcas_filepath: ${OPENSEARCH_PATH_CONF}/certs/root-ca.pemI ask you to check this in your opensearch.yml file and check if the dn is correct.
jplangford
commented
2 years ago Hello again,
Here is the output from the curl command, I think this says that it has loaded the template?
jonathan.langford@dev-wazuh-manager:~$ curl -XGET "https://wazuh.indexer:9200/_cat/templates" -u 'admin:<admin_password>' -k -s
wazuh-statistics [wazuh-statistics-*] 0
wazuh [wazuh-alerts-4.x-*, wazuh-archives-4.x-*] 0 1
wazuh-agent [wazuh-monitoring-*] 0And here is the diff against the opensearch.yml file per your suggestion:
jonathan.langford@dev-wazuh-manager:~$ sudo diff /data/elasticsearch/config/opensearch.yml /data/elasticsearch/config/opensearch.yml.bak
17,19c17,19
< plugins.security.ssl.transport.pemcert_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.pem
< plugins.security.ssl.transport.pemkey_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.key
< plugins.security.ssl.transport.pemtrustedcas_filepath: ${OPENSEARCH_PATH_CONF}/certs/root-ca.pem
---
> plugins.security.ssl.transport.pemcert_filepath: ssl_certs/wazuh.indexer.pem
> plugins.security.ssl.transport.pemkey_filepath: ssl_certs/wazuh.indexer.key
> plugins.security.ssl.transport.pemtrustedcas_filepath: ssl_certs/root-ca.pem
22,24c22,24
< plugins.security.ssl.http.pemcert_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.pem
< plugins.security.ssl.http.pemkey_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.key
< plugins.security.ssl.http.pemtrustedcas_filepath: ${OPENSEARCH_PATH_CONF}/certs/root-ca.pem
---
> plugins.security.ssl.http.pemcert_filepath: ssl_certs/wazuh.indexer.pem
> plugins.security.ssl.http.pemkey_filepath: ssl_certs/wazuh.indexer.key
> plugins.security.ssl.http.pemtrustedcas_filepath: ssl_certs/root-ca.pem
jonathan.langford@dev-wazuh-manager:~$ Unfortunately it is still saying the same thing :(
Here are the DNs from the certificates:
jonathan.langford@dev-wazuh-manager:~$ for cert in `ls /data/elasticsearch/ssl_certs/*.pem`; do echo $cert; openssl x509 -in $cert -noout -text | grep Subject | head -1 ; done
/data/elasticsearch/ssl_certs/admin.pem
Subject: DC = com, DC = acme, O = my_gcp_project_id, OU = DevOps, CN = admin
/data/elasticsearch/ssl_certs/filebeat.pem
Subject: DC = com, DC = acme, O = my_gcp_project_id, OU = DevOps, CN = filebeat
/data/elasticsearch/ssl_certs/root-ca.pem
Subject: DC = com, DC = acme, O = my_gcp_project_id, OU = CA, CN = root-ca
/data/elasticsearch/ssl_certs/wazuh.dashboard.pem
Subject: DC = com, DC = acme, O = my_gcp_project_id, OU = DevOps, CN = wazuh.dashboard
/data/elasticsearch/ssl_certs/wazuh.indexer.pem
Subject: DC = com, DC = acme, O = my_gcp_project_id, OU = DevOps, CN = wazuh.indexer
/data/elasticsearch/ssl_certs/wazuh.manager.pem
Subject: DC = com, DC = acme, O = my_gcp_project_id, OU = DevOps, CN = wazuh.managerHello @jplangford
In the opensearch.yml file I see that you have two indexer nodes configured:
plugins.security.nodes_dn:
-CN=wazuh.indexer,OU=DevOps,O=my_gcp_project_id,DC=acme,DC=com
-CN=filebeat,OU=DevOps,O=my_gcp_project_id,DC=acme,DC=combut you have discovery.type: single-node, is this configuration like this for something special?
I also see that it is missing parameters from the configuration file that we use in the deployment, did you eliminate them for something special? all the configurations that we add to the opensearch.yml file are necessary
Could you try to start your environment with the single-node/config/wazuh_indexer/wazuh.indexer.yml file from thewazuh-docker repository? You should modify the paths of the certificates and the corresponding dn
jplangford
commented
2 years ago Hi @vcerenu ,
That appears to have done the trick. There are no indexing errors in the wazuh-indexer container logs or in the wazuh-manager output and the various screens in the Wazuh UI now actually populate with data. I confess to being entirely perplexed. I have no idea why what you've suggested has fixed it. Here is the opensearch.yml file for future reference:
jonathan.langford@dev-wazuh-manager:~$ cat /data/elasticsearch/config/opensearch.yml
network.host: "0.0.0.0"
node.name: "wazuh.indexer"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer
discovery.type: single-node
compatibility.override_main_response_version: true
plugins.security.ssl.http.pemcert_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.pem
plugins.security.ssl.http.pemkey_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.key
plugins.security.ssl.http.pemtrustedcas_filepath: ${OPENSEARCH_PATH_CONF}/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.pem
plugins.security.ssl.transport.pemkey_filepath: ${OPENSEARCH_PATH_CONF}/certs/wazuh.indexer.key
plugins.security.ssl.transport.pemtrustedcas_filepath: ${OPENSEARCH_PATH_CONF}/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.authcz.admin_dn:
- CN=admin,OU=DevOps,O=my_gcp_project_id,DC=acme,DC=com
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- CN=wazuh.indexer,OU=DevOps,O=my_gcp_project_id,DC=acme,DC=com
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]Hello,
This issue is resolved so I'm closing it. Thank you again for your assistance.
I have confirmed that removing the compatibility.override_main_response_version setting from the opensearch.yml file is sufficient to reproduce this. It appears to be related to https://github.com/opensearch-project/OpenSearch/issues/667 and https://github.com/opensearch-project/OpenSearch/issues/693.
So in summary if you are running containerised Wazuh with the opensearch.yml file mounted from the host filesystem and you see that events are not being indexed then check for the stack traces I've listed in the wazuh.indexer docker container log output.
If you see them then the chances are you've not included the compatibility.override_main_response_version setting in the opensearch.yml.
Thanks
mapper predecoder.timestamp error on 4.3.3
Hi, I am attempting to deploy containerised Wazuh v 4.3.3 and am seeing this error in the wazuh-indexer docker logs output:
I have tried:
The Wazuh UI loads and shows the agents in the main screen but there are no alerts of any kind. Any suggestions gratefully received.
Thank you.
Please find my docker-compose.yml file below:
Wazuh dashboard logs: