search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.16k stars 4.91k forks source link

Winlogbeat v7 to NiFi partial messages #13842

Closed quepasapasha closed 4 years ago

quepasapasha commented 5 years ago

Hi All,

I ship windows events using ListenBeats in NiFi, however some messages arrive truncated. So far I can't identify a pattern why this happens.

what I get on NiFi is this mess: 1

I can see some non-printable characters in the content of arrived message: err2 3

the original event which I can see in Event Viewer has nothing suspicious though:

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          9/26/2019 11:05:45 PM
Event ID:      4627
Task Category: Group Membership
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      xxx.corp.yyyyyyy.ru
Description:
Group membership information.

Subject:
    Security ID:        NULL SID
    Account Name:       -
    Account Domain:     -
    Logon ID:       0x0

Logon Type:         3

New Logon:
    Security ID:        SYSTEM
    Account Name:       zzzzzz01$
    Account Domain:     xxx.corp.yyyyyyy.RU
    Logon ID:       0x14303B0E

Event in sequence:      1 of 1

Group Membership:           
        BUILTIN\Administrators
        Everyone
        BUILTIN\Pre-Windows 2000 Compatible Access
        BUILTIN\Users
        BUILTIN\Windows Authorization Access Group
        NT AUTHORITY\NETWORK
        NT AUTHORITY\Authenticated Users
        NT AUTHORITY\This Organization
        CORP\zzzzz01$
        CORP\Domain Controllers
        NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS
        Authentication authority asserted identity
        CORP\Denied RODC Password Replication Group
        CORP\RAS and IAS Servers
        Mandatory Label\System Mandatory Level

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

This event is generated when the Audit Group Membership subcategory is configured.  The Logon ID field can be used to correlate this event with the corresponding user logon event as well as to any other security audit events generated during this logon session.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
    <EventID>4627</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>12554</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8020000000000000</Keywords>
    <TimeCreated SystemTime="2019-09-26T20:05:45.358484000Z" />
    <EventRecordID>20279178</EventRecordID>
    <Correlation />
    <Execution ProcessID="632" ThreadID="7752" />
    <Channel>Security</Channel>
    <Computer>dc01.corp.kraftlab.ru</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="SubjectUserSid">S-1-0-0</Data>
    <Data Name="SubjectUserName">-</Data>
    <Data Name="SubjectDomainName">-</Data>
    <Data Name="SubjectLogonId">0x0</Data>
    <Data Name="TargetUserSid">S-1-5-18</Data>
    <Data Name="TargetUserName">DC01$</Data>
    <Data Name="TargetDomainName">CORP.KRAFTLAB.RU</Data>
    <Data Name="TargetLogonId">0x14303b0e</Data>
    <Data Name="LogonType">3</Data>
    <Data Name="EventIdx">1</Data>
    <Data Name="EventCountTotal">1</Data>
    <Data Name="GroupMembership">
        %{S-1-5-32-544}
        %{S-1-1-0}
        %{S-1-5-32-554}
        %{S-1-5-32-545}
        %{S-1-5-32-560}
        %{S-1-5-2}
        %{S-1-5-11}
        %{S-1-5-15}
        %{S-1-5-21-900100797-3563623125-4166005187-1000}
        %{S-1-5-21-900100797-3563623125-4166005187-516}
        %{S-1-5-9}
        %{S-1-18-1}
        %{S-1-5-21-900100797-3563623125-4166005187-572}
        %{S-1-5-21-900100797-3563623125-4166005187-553}
        %{S-1-16-16384}</Data>
  </EventData>
</Event>

winlogbeat config winlogbeat.yml :

winlogbeat.event_logs:
  - name: Application
    ignore_older: 168h
    fields_under_root: true
    fields:
      client_name: "region"
    tags: [ "application","windows" ]
  - name: Security
    ignore_older: 168h
    fields_under_root: true
    fields:
      client_name: "region"
    tags: [ "security","windows" ]
    processors:
      - drop_event.when.not.or:
        - range.winlog.event_id: { gte: 4608, lte: 4609 }
        - equals.winlog.event_id: 4616
        - equals.winlog.event_id: 4621
        - range.winlog.event_id: { gte: 4624, lte: 4625 }
        - equals.winlog.event_id: 4627
        - equals.winlog.event_id: 4634
        - range.winlog.event_id: { gte: 4647, lte: 4649 }
        - equals.winlog.event_id: 4656
        - equals.winlog.event_id: 4658
        - range.winlog.event_id: { gte: 4662, lte: 4663 }
        - range.winlog.event_id: { gte: 4672, lte: 4675 }
        - equals.winlog.event_id: 4688
        - equals.winlog.event_id: 4696
        - range.winlog.event_id: { gte: 4698, lte: 4702 }
        - range.winlog.event_id: { gte: 4704, lte: 4707 }
        - range.winlog.event_id: { gte: 4713, lte: 4720 }
        - range.winlog.event_id: { gte: 4722, lte: 4735 }
#        - equals.winlog.event_id: 4724
#        - equals.winlog.event_id: 4728
        - range.winlog.event_id: { gte: 4737, lte: 4764 }
#        - equals.winlog.event_id: 4738
        - equals.winlog.event_id: 4767
        - range.winlog.event_id: { gte: 4774, lte: 4794 }
        - range.winlog.event_id: { gte: 4800, lte: 4803 }
        - range.winlog.event_id: { gte: 4864, lte: 4867 }
        - equals.winlog.event_id: 4902
        - range.winlog.event_id: { gte: 4904, lte: 4908 }
        - equals.winlog.event_id: 4912
        - equals.winlog.event_id: 4964
        - range.winlog.event_id: { gte: 5024, lte: 5025 }
        - range.winlog.event_id: { gte: 5027, lte: 5030 }
        - range.winlog.event_id: { gte: 5032, lte: 5035 }
        - equals.winlog.event_id: 5037
        - range.winlog.event_id: { gte: 5058, lte: 5059 }
        - range.winlog.event_id: { gte: 5136, lte: 5140 }
        - range.winlog.event_id: { gte: 5142, lte: 5144 }
        - range.winlog.event_id: { gte: 5148, lte: 5149 }
        - equals.winlog.event_id: 5168
        - equals.winlog.event_id: 5378
        - range.winlog.event_id: { gte: 5632, lte: 5633 }
        - equals.winlog.event_id: 6416
        - range.winlog.event_id: { gte: 6419, lte: 4624 }
  - name: System
    ignore_older: 168h
    fields_under_root: true
    fields:
      client_name: "region"
    tags: [ "system","windows" ]

output:
  logstash:
    hosts: ["xxxxxxxxxxxxxxxxx:5044"]
    worker: 1
    compression_level: 0
    bulk_max_size: 1
    pipelining: 0
    codec.json:
      pretty: false
      escape_html: true

winlogbeat.shutdown_timeout: 10s
quepasapasha commented 5 years ago

nifi.properties config file:

# Core Properties #
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components

####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# H2 Settings
nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE

# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable.  Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2

# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000

# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

# Site to Site properties
nifi.remote.input.host=
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=

# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=

# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=

# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# nifi.security.identity.mapping.transform.kerb=UPPER

# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER

# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=false

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=false
nifi.cluster.node.address=
nifi.cluster.node.protocol.port=
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #
nifi.kerberos.krb5.file=

# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=

# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
andrewkroh commented 4 years ago

It looks like the outgoing JSON data is getting corrupted in some way. Is is always being corrupted in the same location (I see an event corrupted at the api value). I'd be curious to know if this happens when writing the events to file (rather than logstash). Another thing to try is if it happens in the latest version. If it's still happening in the latest version let us know.

charliek17 commented 4 years ago

@Pavel201909 did you manage to fix this problem?