search

wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
11.11k stars 1.68k forks source link

Error updating feed: [json.exception.out_of_range.403] key 'adp' not found #27060

Open CarlosALgit opened 3 days ago

CarlosALgit commented 3 days ago
Wazuh version Component Install type Install method Platform
4.10.0-beta1 Wazuh Server AIO Installation Assistant AIO Ubuntu 22 / RHEL 9

Description

While doing the Installation Assistant test for v4.10.0-beta1 I got the following error when checking the logs of the manager.

There are some errors that we have already investigated in this issue. The new one is: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found

Ubuntu 22 logs:

These are the logs filtering by errors and warns:

root@ip-172-31-33-56:/home/ubuntu# cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
2024/11/26 12:10:07 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-33-56', retrying until the connection is successful.
2024/11/26 12:13:48 wazuh-modulesd:content-updater: WARNING: The offsets download has been interrupted.
2024/11/26 12:14:02 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-33-56', retrying until the connection is successful.
2024/11/26 12:15:53 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found.
2024/11/26 12:15:53 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.

But then I checked the full logs and looked for the ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found error:

2024/11/26 12:15:32 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/26 12:15:53 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found.
2024/11/26 12:15:53 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
2024/11/26 12:16:17 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/26 12:41:11 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update.
2024/11/26 12:41:11 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed.
2024/11/26 12:41:11 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/26 12:47:26 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update.
2024/11/26 12:47:26 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed.
2024/11/26 13:14:02 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/11/26 13:14:12 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/11/26 14:14:13 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/11/26 14:14:18 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/11/26 15:14:19 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/11/26 15:14:25 wazuh-modulesd:syscollector: INFO: Evaluation finished.
Complete ossec.log ```shellsession root@ip-172-31-33-56:/home/ubuntu# cat /var/ossec/logs/ossec.log 2024/11/26 12:09:56 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:09:56 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:09:59 wazuh-csyslogd: INFO: Remote syslog server not configured. Clean exit. 2024/11/26 12:09:59 wazuh-dbd: INFO: Database not configured. Clean exit. 2024/11/26 12:09:59 wazuh-integratord: INFO: Remote integrations not configured. Clean exit. 2024/11/26 12:09:59 wazuh-agentlessd: INFO: Not configured. Exiting. 2024/11/26 12:09:59 wazuh-authd: INFO: Started (pid: 54861). 2024/11/26 12:09:59 wazuh-authd: INFO: Accepting connections on port 1515. No password required. 2024/11/26 12:09:59 wazuh-authd: INFO: Setting network timeout to 1.000000 sec. 2024/11/26 12:10:00 wazuh-db: INFO: Started (pid: 54877). 2024/11/26 12:10:00 wazuh-db: INFO: Created Global database backup "backup/db/global.db-backup-2024-11-26-12:10:00.gz" 2024/11/26 12:10:01 wazuh-execd: INFO: Started (pid: 54911). 2024/11/26 12:10:02 wazuh-analysisd: INFO: Total rules enabled: '7011' 2024/11/26 12:10:02 wazuh-analysisd: INFO: Started (pid: 54925). 2024/11/26 12:10:03 wazuh-analysisd: INFO: (7200): Logtest started 2024/11/26 12:10:03 wazuh-analysisd: INFO: EPS limit disabled 2024/11/26 12:10:03 wazuh-syscheckd: INFO: Started (pid: 54986). 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6003): Monitoring path: '/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6003): Monitoring path: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6003): Monitoring path: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6003): Monitoring path: '/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key' 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6000): Starting daemon... 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds 2024/11/26 12:10:03 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started. 2024/11/26 12:10:03 rootcheck: INFO: Starting rootcheck scan. 2024/11/26 12:10:03 wazuh-remoted: INFO: Started (pid: 55000). Listening on port 1514/TCP (secure). 2024/11/26 12:10:03 wazuh-remoted: INFO: (1410): Reading authentication keys file. 2024/11/26 12:10:04 wazuh-logcollector: INFO: Monitoring output of command(360): df -P 2024/11/26 12:10:04 wazuh-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d 2024/11/26 12:10:04 wazuh-logcollector: INFO: Monitoring full output of command(360): last -n 20 2024/11/26 12:10:04 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'. 2024/11/26 12:10:04 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/dpkg.log'. 2024/11/26 12:10:04 wazuh-logcollector: INFO: Started (pid: 55034). 2024/11/26 12:10:05 wazuh-monitord: INFO: Started (pid: 55057). 2024/11/26 12:10:06 wazuh-logcollector: INFO: (9203): Monitoring journal entries. 2024/11/26 12:10:06 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:10:06 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:10:06 wazuh-modulesd: INFO: Started (pid: 55079). 2024/11/26 12:10:06 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started. 2024/11/26 12:10:06 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting... 2024/11/26 12:10:06 wazuh-modulesd:osquery: INFO: Module disabled. Exiting... 2024/11/26 12:10:06 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module. 2024/11/26 12:10:06 sca: INFO: Module started. 2024/11/26 12:10:06 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml' 2024/11/26 12:10:06 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started. 2024/11/26 12:10:06 sca: INFO: Starting Security Configuration Assessment scan. 2024/11/26 12:10:06 wazuh-modulesd:router: INFO: Starting router module. 2024/11/26 12:10:06 wazuh-modulesd:database: INFO: Module started. 2024/11/26 12:10:06 wazuh-modulesd:download: INFO: Module started. 2024/11/26 12:10:06 wazuh-modulesd:control: INFO: Starting control thread. 2024/11/26 12:10:06 wazuh-modulesd:content_manager: INFO: Starting content_manager module. 2024/11/26 12:10:06 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml' 2024/11/26 12:10:06 wazuh-modulesd:syscollector: INFO: Module started. 2024/11/26 12:10:06 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 12:10:07 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-33-56', retrying until the connection is successful. 2024/11/26 12:10:07 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 12:10:07 wazuh-modulesd:vulnerability-scanner: INFO: Starting database file decompression. 2024/11/26 12:10:09 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended. 2024/11/26 12:10:09 wazuh-syscheckd: INFO: FIM sync module started. 2024/11/26 12:10:11 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml' 2024/11/26 12:10:11 sca: INFO: Security Configuration Assessment scan finished. Duration: 5 seconds. 2024/11/26 12:10:31 rootcheck: INFO: Ending rootcheck scan. 2024/11/26 12:11:10 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-ip-172-31-33-56. 2024/11/26 12:13:03 wazuh-modulesd:vulnerability-scanner: INFO: Database decompression finished. 2024/11/26 12:13:04 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started. 2024/11/26 12:13:48 wazuh-modulesd:syscollector: INFO: Stop received for Syscollector. 2024/11/26 12:13:48 wazuh-modulesd:syscollector: INFO: Module finished. 2024/11/26 12:13:48 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module. 2024/11/26 12:13:48 wazuh-modulesd:content-updater: WARNING: The offsets download has been interrupted. 2024/11/26 12:13:48 wazuh-modulesd:router: INFO: Stopping router module. 2024/11/26 12:13:48 wazuh-modulesd:content_manager: INFO: Stopping content_manager module. 2024/11/26 12:13:48 wazuh-monitord: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:48 wazuh-logcollector: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:48 wazuh-remoted: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:48 wazuh-syscheckd: INFO: (1756): Shutdown received. Releasing resources. 2024/11/26 12:13:49 wazuh-syscheckd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:49 wazuh-analysisd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:49 wazuh-execd: INFO: (1314): Shutdown received. Deleting responses. 2024/11/26 12:13:49 wazuh-execd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:50 wazuh-db: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:51 wazuh-db: INFO: Graceful process shutdown. 2024/11/26 12:13:51 wazuh-authd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:13:51 wazuh-authd: INFO: Exiting... 2024/11/26 12:13:54 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:13:54 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:13:56 wazuh-csyslogd: INFO: Remote syslog server not configured. Clean exit. 2024/11/26 12:13:56 wazuh-dbd: INFO: Database not configured. Clean exit. 2024/11/26 12:13:56 wazuh-integratord: INFO: Remote integrations not configured. Clean exit. 2024/11/26 12:13:56 wazuh-agentlessd: INFO: Not configured. Exiting. 2024/11/26 12:13:56 wazuh-authd: INFO: Started (pid: 58285). 2024/11/26 12:13:56 wazuh-authd: INFO: Accepting connections on port 1515. No password required. 2024/11/26 12:13:56 wazuh-authd: INFO: Setting network timeout to 1.000000 sec. 2024/11/26 12:13:57 wazuh-db: INFO: Started (pid: 58301). 2024/11/26 12:13:58 wazuh-execd: INFO: Started (pid: 58326). 2024/11/26 12:13:58 wazuh-analysisd: INFO: Total rules enabled: '7011' 2024/11/26 12:13:58 wazuh-analysisd: INFO: Started (pid: 58337). 2024/11/26 12:13:59 wazuh-analysisd: INFO: (7200): Logtest started 2024/11/26 12:13:59 wazuh-analysisd: INFO: EPS limit disabled 2024/11/26 12:13:59 wazuh-syscheckd: INFO: Started (pid: 58398). 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6003): Monitoring path: '/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6003): Monitoring path: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6003): Monitoring path: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6003): Monitoring path: '/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key' 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6000): Starting daemon... 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds 2024/11/26 12:13:59 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started. 2024/11/26 12:13:59 rootcheck: INFO: Starting rootcheck scan. 2024/11/26 12:13:59 wazuh-remoted: INFO: Started (pid: 58412). Listening on port 1514/TCP (secure). 2024/11/26 12:13:59 wazuh-remoted: INFO: (1410): Reading authentication keys file. 2024/11/26 12:14:00 wazuh-logcollector: INFO: Monitoring output of command(360): df -P 2024/11/26 12:14:00 wazuh-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d 2024/11/26 12:14:00 wazuh-logcollector: INFO: Monitoring full output of command(360): last -n 20 2024/11/26 12:14:00 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'. 2024/11/26 12:14:00 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/dpkg.log'. 2024/11/26 12:14:00 wazuh-logcollector: INFO: Started (pid: 58446). 2024/11/26 12:14:00 wazuh-monitord: INFO: Started (pid: 58462). 2024/11/26 12:14:01 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended. 2024/11/26 12:14:01 wazuh-syscheckd: INFO: FIM sync module started. 2024/11/26 12:14:01 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:14:01 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:14:01 wazuh-modulesd: INFO: Started (pid: 58479). 2024/11/26 12:14:01 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started. 2024/11/26 12:14:01 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting... 2024/11/26 12:14:01 wazuh-modulesd:osquery: INFO: Module disabled. Exiting... 2024/11/26 12:14:01 sca: INFO: Module started. 2024/11/26 12:14:01 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml' 2024/11/26 12:14:01 sca: INFO: Starting Security Configuration Assessment scan. 2024/11/26 12:14:01 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module. 2024/11/26 12:14:01 wazuh-modulesd:content_manager: INFO: Starting content_manager module. 2024/11/26 12:14:01 wazuh-modulesd:database: INFO: Module started. 2024/11/26 12:14:01 wazuh-modulesd:router: INFO: Starting router module. 2024/11/26 12:14:01 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started. 2024/11/26 12:14:01 wazuh-modulesd:control: INFO: Starting control thread. 2024/11/26 12:14:01 wazuh-modulesd:download: INFO: Module started. 2024/11/26 12:14:01 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml' 2024/11/26 12:14:01 wazuh-modulesd:syscollector: INFO: Module started. 2024/11/26 12:14:01 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 12:14:01 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 12:14:02 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-33-56', retrying until the connection is successful. 2024/11/26 12:14:02 wazuh-logcollector: INFO: (9203): Monitoring journal entries. 2024/11/26 12:14:02 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started. 2024/11/26 12:14:06 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml' 2024/11/26 12:14:06 sca: INFO: Security Configuration Assessment scan finished. Duration: 5 seconds. 2024/11/26 12:14:24 rootcheck: INFO: Ending rootcheck scan. 2024/11/26 12:15:04 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-ip-172-31-33-56. 2024/11/26 12:15:32 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process. 2024/11/26 12:15:53 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found. 2024/11/26 12:15:53 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download. 2024/11/26 12:16:17 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process. 2024/11/26 12:41:11 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update. 2024/11/26 12:41:11 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed. 2024/11/26 12:41:11 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process. 2024/11/26 12:47:26 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update. 2024/11/26 12:47:26 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed. 2024/11/26 13:14:02 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 13:14:12 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 14:14:13 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 14:14:18 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 15:14:19 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 15:14:25 wazuh-modulesd:syscollector: INFO: Evaluation finished. ```

RHEL 9 logs:

These are the logs filtering by errors and warns:

[root@ip-172-31-34-104 ec2-user]# cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
2024/11/26 12:09:07 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-34-104.ec2.internal', retrying until the connection is successful.
2024/11/26 12:12:11 wazuh-modulesd:content-updater: WARNING: The offsets download has been interrupted.
2024/11/26 12:12:22 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-34-104.ec2.internal', retrying until the connection is successful.
2024/11/26 12:14:13 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found.
2024/11/26 12:14:13 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.

But then I checked the full logs and looked for the ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found error:

2024/11/26 12:13:52 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/26 12:14:13 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found.
2024/11/26 12:14:13 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
2024/11/26 12:14:46 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/26 12:40:51 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update.
2024/11/26 12:40:51 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed.
2024/11/26 12:40:52 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/26 12:49:16 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update.
2024/11/26 12:49:16 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed.
2024/11/26 13:12:23 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/11/26 13:12:33 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/11/26 14:12:34 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/11/26 14:12:41 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/11/26 15:12:42 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/11/26 15:12:49 wazuh-modulesd:syscollector: INFO: Evaluation finished.
Complete ossec.log ```shellsession [root@ip-172-31-34-104 ec2-user]# cat /var/ossec/logs/ossec.log 2024/11/26 12:09:01 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:09:01 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:09:03 wazuh-csyslogd: INFO: Remote syslog server not configured. Clean exit. 2024/11/26 12:09:03 wazuh-dbd: INFO: Database not configured. Clean exit. 2024/11/26 12:09:03 wazuh-integratord: INFO: Remote integrations not configured. Clean exit. 2024/11/26 12:09:03 wazuh-agentlessd: INFO: Not configured. Exiting. 2024/11/26 12:09:04 wazuh-authd: INFO: Started (pid: 17505). 2024/11/26 12:09:04 wazuh-authd: INFO: Accepting connections on port 1515. No password required. 2024/11/26 12:09:04 wazuh-authd: INFO: Setting network timeout to 1.000000 sec. 2024/11/26 12:09:04 wazuh-db: INFO: Started (pid: 17519). 2024/11/26 12:09:04 wazuh-execd: INFO: Started (pid: 17530). 2024/11/26 12:09:04 wazuh-db: INFO: Created Global database backup "backup/db/global.db-backup-2024-11-26-12:09:04.gz" 2024/11/26 12:09:04 wazuh-analysisd: INFO: Total rules enabled: '7011' 2024/11/26 12:09:04 wazuh-analysisd: INFO: Started (pid: 17542). 2024/11/26 12:09:04 wazuh-analysisd: INFO: (7200): Logtest started 2024/11/26 12:09:04 wazuh-analysisd: INFO: EPS limit disabled 2024/11/26 12:09:05 wazuh-syscheckd: INFO: Started (pid: 17616). 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6003): Monitoring path: '/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6003): Monitoring path: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6003): Monitoring path: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6003): Monitoring path: '/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key' 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6000): Starting daemon... 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds 2024/11/26 12:09:05 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started. 2024/11/26 12:09:05 rootcheck: INFO: Starting rootcheck scan. 2024/11/26 12:09:05 wazuh-remoted: INFO: Started (pid: 17631). Listening on port 1514/TCP (secure). 2024/11/26 12:09:05 wazuh-remoted: INFO: (1410): Reading authentication keys file. 2024/11/26 12:09:06 wazuh-logcollector: INFO: Monitoring output of command(360): df -P 2024/11/26 12:09:06 wazuh-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d 2024/11/26 12:09:06 wazuh-logcollector: INFO: Monitoring full output of command(360): last -n 20 2024/11/26 12:09:06 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/audit/audit.log'. 2024/11/26 12:09:06 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'. 2024/11/26 12:09:06 wazuh-logcollector: INFO: Started (pid: 17676). 2024/11/26 12:09:07 wazuh-monitord: INFO: Started (pid: 17695). 2024/11/26 12:09:07 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:09:07 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:09:07 wazuh-modulesd: INFO: Started (pid: 17705). 2024/11/26 12:09:07 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started. 2024/11/26 12:09:07 wazuh-modulesd:router: INFO: Starting router module. 2024/11/26 12:09:07 wazuh-modulesd:osquery: INFO: Module disabled. Exiting... 2024/11/26 12:09:07 wazuh-modulesd:content_manager: INFO: Starting content_manager module. 2024/11/26 12:09:07 sca: INFO: Module started. 2024/11/26 12:09:07 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_rhel9_linux.yml' 2024/11/26 12:09:07 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module. 2024/11/26 12:09:07 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting... 2024/11/26 12:09:07 wazuh-modulesd:database: INFO: Module started. 2024/11/26 12:09:07 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started. 2024/11/26 12:09:07 sca: INFO: Starting Security Configuration Assessment scan. 2024/11/26 12:09:07 wazuh-modulesd:download: INFO: Module started. 2024/11/26 12:09:07 wazuh-modulesd:control: INFO: Starting control thread. 2024/11/26 12:09:07 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_rhel9_linux.yml' 2024/11/26 12:09:07 wazuh-modulesd:syscollector: INFO: Module started. 2024/11/26 12:09:07 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 12:09:07 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-34-104.ec2.internal', retrying until the connection is successful. 2024/11/26 12:09:07 wazuh-modulesd:vulnerability-scanner: INFO: Starting database file decompression. 2024/11/26 12:09:07 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 12:09:08 wazuh-logcollector: INFO: (9203): Monitoring journal entries. 2024/11/26 12:09:10 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended. 2024/11/26 12:09:10 wazuh-syscheckd: INFO: FIM sync module started. 2024/11/26 12:09:12 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_rhel9_linux.yml' 2024/11/26 12:09:12 sca: INFO: Security Configuration Assessment scan finished. Duration: 5 seconds. 2024/11/26 12:09:49 rootcheck: INFO: Ending rootcheck scan. 2024/11/26 12:10:10 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-ip-172-31-34-104.ec2.internal. 2024/11/26 12:11:31 wazuh-modulesd:vulnerability-scanner: INFO: Database decompression finished. 2024/11/26 12:11:34 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started. 2024/11/26 12:12:11 wazuh-modulesd:syscollector: INFO: Stop received for Syscollector. 2024/11/26 12:12:11 wazuh-modulesd:syscollector: INFO: Module finished. 2024/11/26 12:12:11 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module. 2024/11/26 12:12:11 wazuh-modulesd:content-updater: WARNING: The offsets download has been interrupted. 2024/11/26 12:12:11 wazuh-modulesd:router: INFO: Stopping router module. 2024/11/26 12:12:11 wazuh-modulesd:content_manager: INFO: Stopping content_manager module. 2024/11/26 12:12:11 wazuh-monitord: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:12 wazuh-logcollector: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:12 wazuh-remoted: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:12 wazuh-syscheckd: INFO: (1756): Shutdown received. Releasing resources. 2024/11/26 12:12:12 wazuh-syscheckd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:12 wazuh-analysisd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:12 wazuh-execd: INFO: (1314): Shutdown received. Deleting responses. 2024/11/26 12:12:12 wazuh-execd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:12 wazuh-db: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:13 wazuh-db: INFO: Graceful process shutdown. 2024/11/26 12:12:13 wazuh-authd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning... 2024/11/26 12:12:14 wazuh-authd: INFO: Exiting... 2024/11/26 12:12:15 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:12:15 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:12:18 wazuh-csyslogd: INFO: Remote syslog server not configured. Clean exit. 2024/11/26 12:12:18 wazuh-dbd: INFO: Database not configured. Clean exit. 2024/11/26 12:12:18 wazuh-integratord: INFO: Remote integrations not configured. Clean exit. 2024/11/26 12:12:18 wazuh-agentlessd: INFO: Not configured. Exiting. 2024/11/26 12:12:18 wazuh-authd: INFO: Started (pid: 20608). 2024/11/26 12:12:18 wazuh-authd: INFO: Accepting connections on port 1515. No password required. 2024/11/26 12:12:18 wazuh-authd: INFO: Setting network timeout to 1.000000 sec. 2024/11/26 12:12:18 wazuh-db: INFO: Started (pid: 20622). 2024/11/26 12:12:19 wazuh-execd: INFO: Started (pid: 20648). 2024/11/26 12:12:19 wazuh-syscheckd: INFO: Started (pid: 20670). 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6003): Monitoring path: '/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6003): Monitoring path: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6003): Monitoring path: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6003): Monitoring path: '/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'. 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key' 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6000): Starting daemon... 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds 2024/11/26 12:12:19 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started. 2024/11/26 12:12:19 rootcheck: INFO: Starting rootcheck scan. 2024/11/26 12:12:20 wazuh-analysisd: INFO: Total rules enabled: '7011' 2024/11/26 12:12:20 wazuh-analysisd: INFO: Started (pid: 20660). 2024/11/26 12:12:20 wazuh-analysisd: INFO: (7200): Logtest started 2024/11/26 12:12:20 wazuh-analysisd: INFO: EPS limit disabled 2024/11/26 12:12:20 wazuh-remoted: INFO: Started (pid: 20737). Listening on port 1514/TCP (secure). 2024/11/26 12:12:20 wazuh-remoted: INFO: (1410): Reading authentication keys file. 2024/11/26 12:12:21 wazuh-logcollector: INFO: Monitoring output of command(360): df -P 2024/11/26 12:12:21 wazuh-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d 2024/11/26 12:12:21 wazuh-logcollector: INFO: Monitoring full output of command(360): last -n 20 2024/11/26 12:12:21 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/audit/audit.log'. 2024/11/26 12:12:21 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'. 2024/11/26 12:12:21 wazuh-logcollector: INFO: Started (pid: 20772). 2024/11/26 12:12:21 wazuh-monitord: INFO: Started (pid: 20789). 2024/11/26 12:12:21 wazuh-modulesd:router: INFO: Loaded router module. 2024/11/26 12:12:21 wazuh-modulesd:content_manager: INFO: Loaded content_manager module. 2024/11/26 12:12:21 wazuh-modulesd: INFO: Started (pid: 20799). 2024/11/26 12:12:21 wazuh-modulesd:router: INFO: Starting router module. 2024/11/26 12:12:21 wazuh-modulesd:database: INFO: Module started. 2024/11/26 12:12:21 wazuh-modulesd:download: INFO: Module started. 2024/11/26 12:12:21 wazuh-modulesd:osquery: INFO: Module disabled. Exiting... 2024/11/26 12:12:21 wazuh-modulesd:control: INFO: Starting control thread. 2024/11/26 12:12:21 sca: INFO: Module started. 2024/11/26 12:12:21 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_rhel9_linux.yml' 2024/11/26 12:12:21 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started. 2024/11/26 12:12:21 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module. 2024/11/26 12:12:21 wazuh-modulesd:content_manager: INFO: Starting content_manager module. 2024/11/26 12:12:21 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting... 2024/11/26 12:12:21 sca: INFO: Starting Security Configuration Assessment scan. 2024/11/26 12:12:21 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started. 2024/11/26 12:12:21 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_rhel9_linux.yml' 2024/11/26 12:12:21 wazuh-modulesd:syscollector: INFO: Module started. 2024/11/26 12:12:21 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 12:12:21 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended. 2024/11/26 12:12:21 wazuh-syscheckd: INFO: FIM sync module started. 2024/11/26 12:12:22 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 12:12:22 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-34-104.ec2.internal', retrying until the connection is successful. 2024/11/26 12:12:22 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started. 2024/11/26 12:12:23 wazuh-logcollector: INFO: (9203): Monitoring journal entries. 2024/11/26 12:12:26 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_rhel9_linux.yml' 2024/11/26 12:12:26 sca: INFO: Security Configuration Assessment scan finished. Duration: 5 seconds. 2024/11/26 12:13:02 rootcheck: INFO: Ending rootcheck scan. 2024/11/26 12:13:24 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-ip-172-31-34-104.ec2.internal. 2024/11/26 12:13:52 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process. 2024/11/26 12:14:13 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found. 2024/11/26 12:14:13 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download. 2024/11/26 12:14:46 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process. 2024/11/26 12:40:51 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update. 2024/11/26 12:40:51 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed. 2024/11/26 12:40:52 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process. 2024/11/26 12:49:16 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update. 2024/11/26 12:49:16 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed. 2024/11/26 13:12:23 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 13:12:33 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 14:12:34 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 14:12:41 wazuh-modulesd:syscollector: INFO: Evaluation finished. 2024/11/26 15:12:42 wazuh-modulesd:syscollector: INFO: Starting evaluation. 2024/11/26 15:12:49 wazuh-modulesd:syscollector: INFO: Evaluation finished. ```
MiguelazoDS commented 2 days ago

Analysis

I requested the vendor_map information in the Ubuntu machine that logged the error but I couldn't find anything wrong

image

image image

Code review (OUTDATED: This does no longer apply)

Expand - The only place I find an `adp` key is requested is here. https://github.com/wazuh/wazuh/blob/dff4336243f464ddb57dc8124136ec24a0235dbf/src/wazuh_modules/vulnerability_scanner/src/scanOrchestrator/eventDetailsBuilder.hpp#L251 - The lambda that executes that section of the code is called from `EventDetailsBuilder` class. - The mentioned class is executed as part of a chain of responsibility design pattern for `PackageInsert` and `Os` scan types. - Those orchestration are also created during `ReScanSingleAgent` and `ReScanAllAgents` scan types as suborchestrations. - In the run method from ScanOrchestrator class, we decide which of the previous 4 orchestrations will be executed. https://github.com/wazuh/wazuh/blob/5e863f2d711782d429cfcbe3c1ca1c3ef710e7d5/src/wazuh_modules/vulnerability_scanner/src/scanOrchestrator/scanOrchestrator.hpp#L234 - The run method is executed only in processEvent method. https://github.com/wazuh/wazuh/blob/5e863f2d711782d429cfcbe3c1ca1c3ef710e7d5/src/wazuh_modules/vulnerability_scanner/src/scanOrchestrator/scanOrchestrator.hpp#L188 - The processEvent method is executed in for these threads https://github.com/wazuh/wazuh/blob/a064a616408716def34f0a4368396940d2905c82/src/wazuh_modules/vulnerability_scanner/src/vulnerabilityScannerFacade.cpp#L137 https://github.com/wazuh/wazuh/blob/5e863f2d711782d429cfcbe3c1ca1c3ef710e7d5/src/wazuh_modules/vulnerability_scanner/src/scanOrchestrator/scanOrchestrator.hpp#L155 > [!NOTE] > So the error occurs when one of those threads executes the processEvent and the event matches the four types described above. - Starting from the error in the lambda defined in the databaseFeedManager constructor. https://github.com/wazuh/wazuh/blob/74d8be5587a89b0c4c73621adcb3a1fe0a458bb0/src/wazuh_modules/vulnerability_scanner/src/databaseFeedManager/databaseFeedManager.hpp#L319 - The exception is exactly triggered in the postUpdateCallback defined in vulnerabilityScannerFacade.cpp https://github.com/wazuh/wazuh/blob/eb7818710ec933175ecd003327224e8ee47bc393/src/wazuh_modules/vulnerability_scanner/src/vulnerabilityScannerFacade.cpp#L403 - The postUpdateCallback calls a pushEvent that pushes an event in the dispatcher queue mentioned above. https://github.com/wazuh/wazuh/blob/0e88920ea6195895b1e77df9079f340b6b82478c/src/wazuh_modules/vulnerability_scanner/src/vulnerabilityScannerFacade.hpp#L151 > [!NOTE] > But that sequence does not explain to me, how an exception occur during the execution of the postUpdateCallback.

Review installation assistant

We can assume that this issue is platform independent and the issue may be related to the update process itself but I couldn't reproduce it

The starting offset is this. 

20241122003900 ==> 0
20241122005440 ==> 1080537

[!IMPORTANT] After some attempts it was reproduced by restarting the manager right after the database was decompressed. 

2024/11/27 17:35:18 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-jammy.
2024/11/27 17:35:27 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'adp' not found.
2024/11/27 17:35:27 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
2024/11/27 17:35:42 rootcheck: INFO: Ending rootcheck scan.
2024/11/27 17:35:47 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
MiguelazoDS commented 1 day ago

Update

[!NOTE] I'm still unable to find the root cause of this.

curl -s ${repogpg} --max-time 300 --retry 5 --retry-delay 5 --fail | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import DEBIAN_FRONTEND=noninteractive apt-get install wazuh-manager=4.10.0-* -y -q

But the error didn't show up. 

Eventually when trying to reproduce the error I get 

```console
2024/11/28 15:10:22 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.401] array index 56 is out of range.
2024/11/28 15:10:22 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.

We implemented a script to capture the database when the error is found, because it is not that deterministic the occurrence. 

#!/usr/bin/env bash

repogpg="https://packages-dev.wazuh.com/key/GPG-KEY-WAZUH"
wazuh_repo="/etc/apt/sources.list.d/wazuh.list"
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages-dev.wazuh.com/pre-release/apt/ unstable main" > ${wazuh_repo}

curl -s ${repogpg} --max-time 300 --retry 5 --retry-delay 5 --fail | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import
DEBIAN_FRONTEND=noninteractive apt-get install wazuh-manager=4.10.0-* -y -q

systemctl start wazuh-manager.service

# Path to the Wazuh log file
LOG_FILE="/var/ossec/logs/ossec.log"

# Phrase to search for
SEARCH_PHRASE="key 'adp' not found."

echo "Waiting vuldet"
tail -F "$LOG_FILE" | while read -r line; do
   if [[ "$line" == *"Vulnerability scanner module started."* ]]; then
      echo "FOUND"
      sleep 3
      /var/ossec/bin/wazuh-control restart
      echo "EXIT"
      break
   fi
done

# Command to monitor the log file
tail -F "$LOG_FILE" | while read -r line; do
    if [[ "$line" == *"$SEARCH_PHRASE"* ]]; then
        echo "Phrase found: $SEARCH_PHRASE"

        # Get all Wazuh-related processes and kill them
        wazuh_pids=$(pgrep -f wazuh)

        if [[ -n "$wazuh_pids" ]]; then
            echo "Killing Wazuh processes: $wazuh_pids"
            kill -9 $wazuh_pids
        else
            echo "No Wazuh processes found to kill."
        fi

        # Optional: exit monitoring after killing processes
        break
    fi
done

# Adjust this command.
rm -rf queue; cp -r /var/ossec/queue/ .

And another error was found 

2024/11/28 16:54:33 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
2024/11/28 16:54:35 wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.403] key 'versions' not found.
2024/11/28 16:54:35 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
MiguelazoDS commented 1 day ago

Update

The error was captured and a database backup was performed 

Started wazuh-modulesd...
Completed.
EXIT
Phrase found: key 'adp' not found.
Killing Wazuh processes: 1763408
1763409
1763412
1763415
1763456
1763472
1763509
1763535
1763642
1763656
1763695
1763714
1763723

[!NOTE] Although I don't have still the reason for this behavior, it's clear that the error does not occur in the eventDetailsBuilder. This error message is related to the "adp" in the CVE5 message during an upgrade. That's why we also get "versions" ocasionally.

MiguelazoDS commented 12 hours ago

Update

Discussing this with the team, we suspect that the "adp" and "versions" error mentioned above are related to the JSON patch. A similar case is easily reproduced defining a json object 

{"key":"value"}

and a json patch 

  [
    {
        "op": "replace",
        "path": "/key1",
        "value": "another_value"
    }
    ]
terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_3::detail::out_of_range'
  what():  [json.exception.out_of_range.403] key 'key1' not found
[1]    1150038 IOT instruction (core dumped)  ./a.out

And checking the logs where a invalid index error occurs 

2024/11/29 10:33:29 wazuh-modulesd:vulnerability-scanner[71493] databaseFeedManager.hpp:353 at operator()(): INFO: Initiating update feed process.
2024/11/29 10:33:29 wazuh-modulesd:vulnerability-scanner[71493] databaseFeedManager.hpp:187 at processMessage(): DEBUG: Processing file: queue/vd_updater/tmp/contents/1098537-api_file.json
2024/11/29 10:33:31 wazuh-modulesd:vulnerability-scanner[71493] databaseFeedManager.hpp:374 at operator()(): ERROR: Error updating feed: [json.exception.out_of_range.401] array index 6 is out of range.

We can see that the content updater attempts to process a corrupted file.

sebasfalcone commented 7 hours ago

Testing

We simulated the following scenario:

We simulated this by coping the same file twice in the content:

root@sebas-ThinkPad:/var/ossec/queue/vd_updater/tmp/contents# diff 1103228-api_file.json 1104228-api_file.json

Once the second file is being processed, the exception arises:

2024/11/29 17:03:16 wazuh-modulesd:vulnerability-scanner[28088] databaseFeedManager.hpp:187 at processMessage(): DEBUG: Processing file: queue/vd_updater/tmp/contents/1103228-api_file.json
2024/11/29 17:03:39 wazuh-modulesd:vulnerability-scanner[28088] databaseFeedManager.hpp:187 at processMessage(): DEBUG: Processing file: queue/vd_updater/tmp/contents/1104228-api_file.json
2024/11/29 17:03:40 wazuh-modulesd:vulnerability-scanner[28088] databaseFeedManager.hpp:370 at operator()(): ERROR: Error updating feed: [json.exception.out_of_range.401] array index 6 is out of range.

The most common exception is an out of range one, because most offsets contain changes over the "affected" array

Update

I was able to replicate this "naturally" while using gdb and calling for a stop of all services, because gdb wasn't releasing the process it took a lot of time to process the signal. Once gdb was stopped, modules terminated the offset never updated, this triggered the failure condition

2024/11/29 20:11:50 wazuh-modulesd:vulnerability-scanner[103348] databaseFeedManager.hpp:352 at operator()(): INFO: Initiating update feed process.
2024/11/29 20:11:50 wazuh-modulesd:vulnerability-scanner[103348] databaseFeedManager.hpp:187 at processMessage(): DEBUG: Processing file: queue/vd_updater/tmp/contents/1114228-api_file.json
2024/11/29 20:11:52 wazuh-modulesd:vulnerability-scanner[103348] databaseFeedManager.hpp:370 at operator()(): ERROR: Error updating feed: [json.exception.out_of_range.403] key 'descriptions' not found.

2024/11/29 20:32:47 wazuh-modulesd:vulnerability-scanner[113985] databaseFeedManager.hpp:352 at operator()(): INFO: Initiating update feed process.
2024/11/29 20:32:47 wazuh-modulesd:vulnerability-scanner[113985] databaseFeedManager.hpp:187 at processMessage(): DEBUG: Processing file: queue/vd_updater/tmp/contents/1103228-api_file.json
2024/11/29 20:32:48 wazuh-modulesd:vulnerability-scanner[113985] databaseFeedManager.hpp:370 at operator()(): ERROR: Error updating feed: [json.exception.out_of_range.401] array index 6 is out of range.
MiguelazoDS commented 6 hours ago

Update

Failing update

It was finally possible to reproduce the same error mentioned in the issue description and the reason of that. 

#include <nlohmann/json.hpp>

const std::string jsonToPatch{R"(
{"containers":{"cna":{"affected":[{"cpes":["cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:a2:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:a3:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:a4:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:a5:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:beta1:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:beta2:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:beta3:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:beta4:*:*:*:*:*:*","cpe:2.3:a:c-blosc2_project:c-blosc2:2.0.0:beta5:*:*:*:*:*:*"],"defaultStatus":"unaffected","product":"c-blosc2","vendor":"c-blosc2_project","versions":[{"status":"affected","version":"2.0.0"}]}],"descriptions":[{"lang":"en","value":"blosc2.c in Blosc C-Blosc2 through 2.0.0.beta.5 has a heap-based buffer overflow when there is a lack of space to write compressed data."},{"lang":"es","value":"El archivo blosc2.c en Blosc C-Blosc2 hasta versión 2.0.0.beta.5. presenta un desbordamiento de búfer en la región heap de la memoria cuando carece de espacio para escribir datos comprimidos"}],"metrics":[{"cvssV2_0":{"accessComplexity":"MEDIUM","accessVector":"NETWORK","authentication":"NONE","availabilityImpact":"COMPLETE","baseScore":9.3,"confidentialityImpact":"COMPLETE","environmentalScore":0.0,"integrityImpact":"COMPLETE","temporalScore":0.0,"vectorString":"AV:N/AC:M/Au:N/C:C/I:C/A:C","version":"2.0"},"format":"CVSS"},{"cvssV3_1":{"attackComplexity":"LOW","attackVector":"LOCAL","availabilityImpact":"HIGH","baseScore":7.8,"baseSeverity":"HIGH","confidentialityImpact":"HIGH","environmentalScore":0.0,"integrityImpact":"HIGH","privilegesRequired":"NONE","scope":"UNCHANGED","temporalScore":0.0,"userInteraction":"REQUIRED","vectorString":"CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H","version":"3.1"},"format":"CVSS"}],"problemTypes":[{"descriptions":[{"cweId":"CWE-787","description":"CWE-787","lang":"en"}]}],"providerMetadata":{"dateUpdated":"2020-12-03T20:58:00Z","orgId":"00000000-0000-4000-A000-000000000003","shortName":"nvd","x_subShortName":"nvd"},"references":[{"tags":["mailing-list","third-party-advisory"],"url":"https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26442"},{"tags":["patch","third-party-advisory"],"url":"https://github.com/Blosc/c-blosc2/commit/c4c6470e88210afc95262c8b9fcc27e30ca043ee"}]}},"cveMetadata":{"assignerOrgId":"8254265b-2729-46b6-b9e3-3dfca2d5bfca","assignerShortName":"mitre","cveId":"CVE-2020-29367","datePublished":"2020-11-27T20:15:11Z","dateUpdated":"2020-12-03T20:58:00Z","serial":0,"state":"PUBLISHED"},"dataType":"CVE_RECORD","dataVersion":"5.0"}
)"};

const std::string jsonPatch{R"(
[{"op":"remove","path":"/containers/adp/0/metrics"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/51"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/50"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/49"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/48"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/47"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/46"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/45"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/44"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/43"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/42"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/41"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/40"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/39"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/38"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/37"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/36"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/35"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/34"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/33"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/32"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/31"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/30"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/29"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/28"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/27"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/26"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/25"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/24"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/23"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/22"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/21"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/20"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/19"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/18"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/17"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/16"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/15"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/14"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/13"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/12"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/11"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/10"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/9"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/8"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/7"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/6"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/5"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/4"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/3"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/2"},{"op":"remove","path":"/containers/adp/0/affected/0/versions/1"}]
)"};

int main(int argc, char** argv) {
  nlohmann::json cve5 = nlohmann::json::parse(jsonToPatch);
  nlohmann::json patch = nlohmann::json::parse(jsonPatch);

  cve5.patch_inplace(patch);
  return 0;
}

image

The CVE that receives that patch does not have adp information (CVE-2020-29367), but the previously processed CVE does (CVE-2020-29363)

Log file

image ossec.tar.gz

Offsets written in the database

20241122003900 ==> 0
20241122005440 ==> 1080537
20241129203150 ==> 1091537
20241129203255 ==> 1101537
sebasfalcone commented 3 hours ago

Conclusion

Based on the investigation done with @MiguelazoDS I conclude that the issue concerns the graceful shutdown mechanism. If for some reason took more than expected, the module will be terminated and the offset never updated

The same offset will be processed and in many cases fail (depending on the patch to be applied, deletions are the only cases I was able to use to replicate this failure)

I believe the fix to this is not worth the effort and it will require changing how we handle shutdowns. Note that this scenario is not common and involves two things:

[!IMPORTANT] The module is able to recover from this by processing a snapshot