elasticmachine
commented
3 years ago Pinging @elastic/obs-docs (Team:Docs)
Closed stefnestor closed 3 years ago
elasticmachine
commented
3 years ago Pinging @elastic/obs-docs (Team:Docs)
elasticmachine
commented
3 years ago Pinging @elastic/ingest-management (Team:Ingest Management)
stefnestor
commented
3 years ago Maybe those are the appropriate labels? (Kindly correct if not, thanks!)
stefnestor
commented
3 years ago Edit: the Doc auditbeat.reference.yml is also different than the auditbeat.reference.yml that downloads, guess I needed to look even closer, sorry.
andrewkroh
commented
3 years ago my download is missing the /modules.d/*.yml folder. The Auditbeat Github references modules/system under the x-pack folder.
Auditbeat supports module config loading as indicated in the reference.yml, but there are so few modules (three in the Elastic Licensed distro) that it includes all their config in the main config file (and no modules.d included in the artifacts). You could separate your module config from the main config if that suits you though.
your Configure modules should still reference module: auditd which isn't under the downloaded auditbeat.yml but references under the Filebeat code folder.
auditd is a Linux only feature so you will only see that included in the auditbeat.yml if you download a Linux artifact.
auditbeat.reference.yml doesn't match close to the downloaded un-edited auditbeat.yml
The reference configs generally show every available option with more verbose descriptions. There's no set rule, but the non-reference configs usually contain the most commonly used options with good config values to get started with. In Auditbeat these are tailored to the target OS and architecture.
elasticmachine
commented
3 years ago Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
andrewkroh
commented
3 years ago I think this is not a bug so I'm closing it, but if you have more questions don't hesitate to ask.
Howdy!
I may not be understanding, but your downloaded & Docs auditbeat.reference.yml doesn't match close to the downloaded un-edited auditbeat.yml. Isn't it suppose to? (It does on the Filebeat & Packetbeat beats.)
In particular, I'm wondering if
/modules.d/*.ymlfolder. The Auditbeat Github references modules/system under the x-pack folder.module: auditdwhich isn't under the downloaded auditbeat.yml but references under the Filebeat code folder.Downloaded here to Mac today reporting as v7.12.1.
Thanks in advance!
Stef
The auditbeat.yml and auditbeat.reference.yml's ...
Downloaded
Full
``` ###################### Auditbeat Configuration Example ######################### # This is an example configuration file highlighting only the most common # options. The auditbeat.reference.yml file from the same directory contains all # the supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/auditbeat/index.html # =========================== Modules configuration ============================ auditbeat.modules: - module: file_integrity paths: - /bin - /usr/bin - /usr/local/bin - /sbin - /usr/sbin - /usr/local/sbin - module: system datasets: - package # Installed, updated, and removed packages period: 2m # The frequency at which the datasets check for changes - module: system datasets: - host # General host information, e.g. uptime, IPs - process # Started and stopped processes # How often datasets send state updates with the # current state of the system (e.g. all currently # running processes, all open sockets). state.period: 12h # ======================= Elasticsearch template setting ======================= setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false # ================================== General =================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging # ================================= Dashboards ================================= # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here or by using the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: # =================================== Kibana =================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" # Kibana Space ID # ID of the Kibana Space into which the dashboards should be loaded. By default, # the Default Space will be used. #space.id: # =============================== Elastic Cloud ================================ # These settings simplify using Auditbeat with the Elastic Cloud (https://cloud.elastic.co/). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `Reference
Doc reference link
Full
``` ########################## Auditbeat Configuration ############################# # This is a reference configuration file documenting all non-deprecated options # in comments. For a shorter configuration example that contains only the most # common options, please see auditbeat.yml in the same directory. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/auditbeat/index.html # ============================== Config Reloading ============================== # Config reloading allows to dynamically load modules. Each file which is # monitored must contain one or multiple modules as a list. auditbeat.config.modules: # Glob pattern for configuration reloading path: ${path.config}/modules.d/*.yml # Period on which files under path should be checked for changes reload.period: 10s # Set to true to enable config reloading reload.enabled: false # Maximum amount of time to randomly delay the start of a dataset. Use 0 to # disable startup delay. auditbeat.max_start_delay: 10s # =========================== Modules configuration ============================ auditbeat.modules: # The file integrity module sends events when files are changed (created, # updated, deleted). The events contain file metadata and hashes. - module: file_integrity paths: - /bin - /usr/bin - /usr/local/bin - /sbin - /usr/sbin - /usr/local/sbin # List of regular expressions to filter out notifications for unwanted files. # Wrap in single quotes to workaround YAML escaping rules. By default no files # are ignored. exclude_files: - '\.DS_Store$' - '\.swp$' # List of regular expressions used to explicitly include files. When configured, # Auditbeat will ignore files unless they match a pattern. #include_files: #- '/\.ssh($|/)' # Scan over the configured file paths at startup and send events for new or # modified files since the last time Auditbeat was running. scan_at_start: true # Average scan rate. This throttles the amount of CPU and I/O that Auditbeat # consumes at startup while scanning. Default is "50 MiB". scan_rate_per_sec: 50 MiB # Limit on the size of files that will be hashed. Default is "100 MiB". # Limit on the size of files that will be hashed. Default is "100 MiB". max_file_size: 100 MiB # Hash types to compute when the file changes. Supported types are # blake2b_256, blake2b_384, blake2b_512, md5, sha1, sha224, sha256, sha384, # sha512, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384, sha3_512, and xxh64. # Default is sha1. hash_types: [sha1] # Detect changes to files included in subdirectories. Disabled by default. recursive: false # Set to true to publish fields with null values in events. #keep_null: false # The system module collects security related information about a host. # All datasets send both periodic state information (e.g. all currently # running processes) and real-time changes (e.g. when a new process starts # or stops). - module: system datasets: - package # Installed, updated, and removed packages period: 2m # The frequency at which the datasets check for changes - module: system datasets: - host # General host information, e.g. uptime, IPs - process # Started and stopped processes # How often datasets send state updates with the # current state of the system (e.g. all currently # running processes, all open sockets). state.period: 12h # The state.period can be overridden for any dataset. # host.state.period: 12h # package.state.period: 12h # process.state.period: 12h # Average file read rate for hashing of the process executable. Default is "50 MiB". process.hash.scan_rate_per_sec: 50 MiB # Limit on the size of the process executable that will be hashed. Default is "100 MiB". process.hash.max_file_size: 100 MiB # Hash types to compute of the process executable. Supported types are # blake2b_256, blake2b_384, blake2b_512, md5, sha1, sha224, sha256, sha384, # sha512, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384, sha3_512, and xxh64. # Default is sha1. process.hash.hash_types: [sha1] # ================================== General =================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. # If this options is not defined, the hostname is used. #name: # The tags of the shipper are included in their own field with each # transaction published. Tags make it easy to group servers by different # logical properties. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. Fields can be scalar values, arrays, dictionaries, or any nested # combination of these. #fields: # env: staging # If this option is set to true, the custom fields are stored as top-level # fields in the output document instead of being grouped under a fields # sub-dictionary. Default is false. #fields_under_root: false # Internal queue configuration for buffering events to be published. #queue: # Queue type by name (default 'mem') # The memory queue will present all available events (up to the outputs # bulk_max_size) to the output, the moment the output is ready to server # another batch of events. #mem: # Max number of events the queue can buffer. #events: 4096 # Hints the minimum number of events stored in the queue, # before providing a batch of events to the outputs. # The default value is set to 2048. # A value of 0 ensures events are immediately available # to be sent to the outputs. #flush.min_events: 2048 # Maximum duration after which events are available to the outputs, # if the number of events stored in the queue is < `flush.min_events`. #flush.timeout: 1s # The disk queue stores incoming events on disk until the output is # ready for them. This allows a higher event limit than the memory-only # queue and lets pending events persist through a restart. #disk: # The directory path to store the queue's data. #path: "${path.data}/diskqueue" # The maximum space the queue should occupy on disk. Depending on # input settings, events that exceed this limit are delayed or discarded. #max_size: 10GB # The maximum size of a single queue data file. Data in the queue is # stored in smaller segments that are deleted after all their events # have been processed. #segment_size: 1GB # The number of events to read from disk to memory while waiting for # the output to request them. #read_ahead: 512 # The number of events to accept from inputs while waiting for them # to be written to disk. If event data arrives faster than it # can be written to disk, this setting prevents it from overflowing # main memory. #write_ahead: 2048 # The duration to wait before retrying when the queue encounters a disk # write error. #retry_interval: 1s # The maximum length of time to wait before retrying on a disk write # error. If the queue encounters repeated errors, it will double the # length of its retry interval each time, up to this maximum. #max_retry_interval: 30s # The spool queue will store events in a local spool file, before # forwarding the events to the outputs. # # Beta: spooling to disk is currently a beta feature. Use with care. # # The spool file is a circular buffer, which blocks once the file/buffer is full. # Events are put into a write buffer and flushed once the write buffer # is full or the flush_timeout is triggered. # Once ACKed by the output, events are removed immediately from the queue, # making space for new events to be persisted. #spool: # The file namespace configures the file path and the file creation settings. # Once the file exists, the `size`, `page_size` and `prealloc` settings # will have no more effect. #file: # Location of spool file. The default value is ${path.data}/spool.dat. #path: "${path.data}/spool.dat" # Configure file permissions if file is created. The default value is 0600. #permissions: 0600 # File size hint. The spool blocks, once this limit is reached. The default value is 100 MiB. #size: 100MiB # The files page size. A file is split into multiple pages of the same size. The default value is 4KiB. #page_size: 4KiB # If prealloc is set, the required space for the file is reserved using # truncate. The default value is true. #prealloc: true # Spool writer settings # Events are serialized into a write buffer. The write buffer is flushed if: # - The buffer limit has been reached. # - The configured limit of buffered events is reached. # - The flush timeout is triggered. #write: # Sets the write buffer size. #buffer_size: 1MiB # Maximum duration after which events are flushed if the write buffer # is not full yet. The default value is 1s. #flush.timeout: 1s # Number of maximum buffered events. The write buffer is flushed once the # limit is reached. #flush.events: 16384 # Configure the on-disk event encoding. The encoding can be changed # between restarts. # Valid encodings are: json, ubjson, and cbor. #codec: cbor #read: # Reader flush timeout, waiting for more events to become available, so # to fill a complete batch as required by the outputs. # If flush_timeout is 0, all available events are forwarded to the # outputs immediately. # The default value is 0s. #flush.timeout: 0s # Sets the maximum number of CPUs that can be executing simultaneously. The # default is the number of logical CPUs available in the system. #max_procs: # ================================= Processors ================================= # Processors are used to reduce the number of fields in the exported event or to # enhance the event with external metadata. This section defines a list of # processors that are applied one by one and the first one receives the initial # event: # # event -> filter1 -> event1 -> filter2 ->event2 ... # # The supported processors are drop_fields, drop_event, include_fields, # decode_json_fields, and add_cloud_metadata. # # For example, you can use the following processors to keep the fields that # contain CPU load percentages, but remove the fields that contain CPU ticks # values: # #processors: # - include_fields: # fields: ["cpu"] # - drop_fields: # fields: ["cpu.user", "cpu.system"] # # The following example drops the events that have the HTTP response code 200: # #processors: # - drop_event: # when: # equals: # http.code: 200 # # The following example renames the field a to b: # #processors: # - rename: # fields: # - from: "a" # to: "b" # # The following example tokenizes the string into fields: # #processors: # - dissect: # tokenizer: "%{key1} - %{key2}" # field: "message" # target_prefix: "dissect" # # The following example enriches each event with metadata from the cloud # provider about the host machine. It works on EC2, GCE, DigitalOcean, # Tencent Cloud, and Alibaba Cloud. # #processors: # - add_cloud_metadata: ~ # # The following example enriches each event with the machine's local time zone # offset from UTC. # #processors: # - add_locale: # format: offset # # The following example enriches each event with docker metadata, it matches # given fields to an existing container id and adds info from that container: # #processors: # - add_docker_metadata: # host: "unix:///var/run/docker.sock" # match_fields: ["system.process.cgroup.id"] # match_pids: ["process.pid", "process.ppid"] # match_source: true # match_source_index: 4 # match_short_id: false # cleanup_timeout: 60 # labels.dedot: false # # To connect to Docker over TLS you must specify a client and CA certificate. # #ssl: # # certificate_authority: "/etc/pki/root/ca.pem" # # certificate: "/etc/pki/client/cert.pem" # # key: "/etc/pki/client/cert.key" # # The following example enriches each event with docker metadata, it matches # container id from log path available in `source` field (by default it expects # it to be /var/lib/docker/containers/*/*.log). # #processors: # - add_docker_metadata: ~ # # The following example enriches each event with host metadata. # #processors: # - add_host_metadata: ~ # # The following example enriches each event with process metadata using # process IDs included in the event. # #processors: # - add_process_metadata: # match_pids: ["system.process.ppid"] # target: system.process.parent # # The following example decodes fields containing JSON strings # and replaces the strings with valid JSON objects. # #processors: # - decode_json_fields: # fields: ["field1", "field2", ...] # process_array: false # max_depth: 1 # target: "" # overwrite_keys: false # #processors: # - decompress_gzip_field: # from: "field1" # to: "field2" # ignore_missing: false # fail_on_error: true # # The following example copies the value of message to message_copied # #processors: # - copy_fields: # fields: # - from: message # to: message_copied # fail_on_error: true # ignore_missing: false # # The following example truncates the value of message to 1024 bytes # #processors: # - truncate_fields: # fields: # - message # max_bytes: 1024 # fail_on_error: false # ignore_missing: true # # The following example preserves the raw message under event.original # #processors: # - copy_fields: # fields: # - from: message # to: event.original # fail_on_error: false # ignore_missing: true # - truncate_fields: # fields: # - event.original # max_bytes: 1024 # fail_on_error: false # ignore_missing: true # # The following example URL-decodes the value of field1 to field2 # #processors: # - urldecode: # fields: # - from: "field1" # to: "field2" # ignore_missing: false # fail_on_error: true # =============================== Elastic Cloud ================================ # These settings simplify using Auditbeat with the Elastic Cloud (https://cloud.elastic.co/). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `