Windows sidecar sends variables instead of data from one host.

[hugalafutro]

Problem description

I have 2 Windows 11 machines with sidecar installed with identical config. Both worked as expected, but recently (can't pinpoint the exact time) I noticed one of them sends all meaningful data as %1, %2, %3 etc instead.

example:

2022-10-07 11:38:28.684 | turbojeba
A user's local group membership was enumerated.  Subject: Security ID: %4 Account Name: %5 Account Domain: %6 Logon ID: %7  User: Security ID: %3 Account Name: %1 Account Domain: %2  Process Information: Process ID: %8 Process Name: %9

Steps to reproduce the problem

1. no idea I just noticed it is happening on one machine

Environment

• Sidecar Version: 1.2.0
• Graylog Version: 4.3.8+8c4705e (docker)
• Operating System: Ubuntu 22.04.1 vm
• Elasticsearch Version: 7.10.2 (docker)
• MongoDB Version: 4.2 (docker)

winlogbeat config on graylog side:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["192.168.1.249:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security5140

sidecar.yml from the offending machine:

# The URL to the Graylog server API.
# Default: "http://127.0.0.1:9000/api/"
server_url: "http://192.168.1.249:9000/api"

# The API token to use to authenticate against the Graylog server API.
# Default: none
server_api_token: "[REDACTED]"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
# Default: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
# Default: ""
node_name: "turbojeba"

# The update interval in secods. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
# Default: 10
update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: false

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
# Default: true
send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []

# Directory where the sidecar stores internal data.
#cache_path: "C:\\Program Files\\Graylog\\sidecar\\cache"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "C:\\Program Files\\Graylog\\sidecar\\generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_accesslist:
#       - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#       - "C:\\Program Files\\Filebeat\\filebeat.exe"
#
# Example disable access listing:
#     collector_binaries_accesslist: []
#
# Default:
# collector_binaries_accesslist:
#  - "C:\\Program Files\\Graylog\\sidecar\\filebeat.exe"
#  - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#  - "C:\\Program Files\\Filebeat\\filebeat.exe"
#  - "C:\\Program Files\\Packetbeat\\packetbeat.exe"
#  - "C:\\Program Files\\Metricbeat\\metricbeat.exe"
#  - "C:\\Program Files\\Heartbeat\\heartbeat.exe"
#  - "C:\\Program Files\\Auditbeat\\auditbeat.exe"
#  - "C:\\Program Files (x86)\\nxlog\\nxlog.exe"

[bernd]

@Graylog2/seccon This is probably because of the old winlogbeat version we ship in sidecar. (7.11) Can you please check if you can reproduce the issue? Thank you!

[hugalafutro]

For what it's worth, both machines were using sidecar 1.1.0 for the longest time, survived update from win10 to win11 etc. Then I noticed this problem I'd say between a week and month ago, and updated sidecar on both machines to 1.2.0 with no change in behaviour just before posting this issue. Winlogbeat.exe is 7.11.1 on both the working and non-working machine.

[hugalafutro]

it seems the Windows 11 update 22H2 is the culprit, the working PC just finished updating to it and after reboot sends the messages with %1 %2 %3 etc now too.

[drewmiranda-gl]

Confirming i see the same behavior with Windows 11 22H2.

For what its worth i can still replicate this issue using the latest version of Winlogbeat, 8.5.0.

I did some cursory searching on the internet and cannot find any acknowledgement from elastic that this issue exists. I did post on their community forum to see if its possible a fix is in the works but just not publicly disclosed.

[hugalafutro]

After updating some components the issue is still replicable in Graylog 5.0.0-rc.2+6a55921, with Sidecars 1.2.0 and 1.3.0-beta.1

[bernd]

@hugalafutro The issue is not related to Graylog or Sidecar. There seems to be an issue with winlogbeat and newer Windows versions. This is nothing we can fix in our products.

[hugalafutro]

@bernd Thanks for letting me know, am I correct in assuming I should report the issue at https://github.com/elastic/beats/issues then?

[bernd]

A user's local group membership was enumerated. Subject: Security ID: %4

@hugalafutro Please try to update to the latest winlogbeat first. The issue might be fixed already. :slightly_smiling_face: The version we ship in our Sidecar for Windows is outdated. We are looking into updating it soon.

[hugalafutro]

@bernd I have tried with winlogbeat 8.5.2 released 7 days ago, but the issue is the same, I shall open the issue at elastic later (gotta do laundry first 😄)

for reference this is an example event from graylog with winlogbeat 8.5.2 https://o.o5.ddns.net/8x90h

I'm not sure if you want me to close this or leave it open until it is actually resolved even though it is not actually graylog issue, let me know.

edit: I have found the issue at elastic, although only in their unconfirmed bugs section https://discuss.elastic.co/t/winlogbeat-8-5-and-windows-11-22h2/318676/3

[drewmiranda-gl]

Confirmed that winlogbeat 8.6.0 does not resolve this issue.

[mpfz0r]

For reference, the issue at elastic https://github.com/elastic/beats/issues/33966

[bernd]

@mpfz0r This is most likely a bug in Windows 11 22H2. We are in contact with Microsoft about it.

[efd6]

@bernd I have just done some digging in winlogbeat to follow what is happening here and am fairly confident that this is a Windows 11 issue (summary of investigation here). Have you had any response from MS?

[bernd]

@bernd I have just done some digging in winlogbeat to follow what is happening here and am fairly confident that this is a Windows 11 issue (summary of investigation here). Have you had any response from MS?

@efd6 I agree. This looks like a Windows 11 issue to me as well. It works correctly with Windows 10 22H2 and Windows 11 21H2. It breaks since Windows 11 22H2.

We don't have a response from Microsoft yet.

[drewmiranda-gl]

Quick verification check, are y'all able to reproduce this issue? I just tried with Windows 11 10.0.19044.2728 (March 2023 patch level) and no longer experience this issue. Want to confirm though.

Ignore, i was confused and wrong :(

[bernd]

Quick verification check, are y'all able to reproduce this issue? I just tried with Windows 11 10.0.19044.2728 (March 2023 patch level) and no longer experience this issue. Want to confirm though.

My last test was in February. So they might have fixed the issue in a newer version. That would be great. :slightly_smiling_face:

[drewmiranda-gl]

Ok, i really messed that testing up 🫠. It had been so long that I forgot specifically which windows device I was testing with and then re-tested with a windows 10 device that never had this issue.

So to clarify, the issue still exists, and still occurs with the latest Winlogbeat 8.7.1. Sorry for the confusion!

[ernie-sys]

Any solution for the problem?

[drewmiranda-gl]

Via https://github.com/elastic/beats/issues/33966#issuecomment-1570019443

Microsoft have advised us that what we're seeing is a known issue and was addressed in an update released on May 24th - KB5026446. The update is currently an optional update, but will be included in next month's Patch Tuesday.

I independently tested this and can confirm it does resolve this issue!

[mpfz0r]

Can we close this issue then? Not a bug on our side and MS provided a fix

[drewmiranda-gl]

We can. Bug on microsoft side and took them ~8 months to fix 🙃