2024-01-16

[so3500]

Filebeat

• Filebeat quick start: installation and configuration
• Configure > Inputs > Log
• Configure > Inputs > filestream
• Configure > Output > Kafka

[so3500]

[ configura the kafka output ]

https://www.elastic.co/guide/en/beats/filebeat/7.17/kafka-output.html

hosts

• The kafka topic used for produced events.
• you can set the topic dynamically by using a format string to access any event field. for example, this configuration uses a custim field, fields.log_topic, to set the topic for each event
• topic: '%{[fields.log_topic]}' topic
• That list of Kafka broker addresses from where to fetch the cluster metadata. The cluster metadata contain the actual Kafka brokers events are published to. required_acks
• The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1
• If set to 0, no ACKs are returned by Kafka. Messges might be lost silently on error. ussername
• The ussername for connecting to Kafka. If username is configured, the password must be configured as well. password
• The password for connecting to Kafka. sasl.mechanism
• The SASL mechanism to use when connecting to Kafka. It can be one of:
  • PLAIN for SASL/PLAIN
  • SCRAM-SHA-256 for SCRAM-SHA-256
  • SCRAM-SHA-512 for SCRAM-SHA-512
  • If sasl.mechainsm is not set, PLAIN is used if username and password are provided. Otherwise, SASL authentication is disabled.

[ Configure > Inputs > Log ]

https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-log.html

The log input is deprecated. Please use the filestream input for sending log files to ouptut

Use the log input to read lines from log files. To configuration this input, specify a list of glob-based paths that must be crawled to locate and fetch the log lines.

example configuration:

filebeat.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log

you can apply additional configuraton settings (such as fileds, include_lines, exclude_lines, multiline, ...)

configurations

tail_files

• https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-log.html#_tail_files_3
• If this option is set to true, Filebeat starts reading new files at the end of each file instead of the beginning.
• When this option is used in combination with log rotation, It's possible that the first log entries in a new file might be skipped.
• The default setting is false.
• This option applies to files that Filebeat has not already processed. If you ran Filebeat previously and the state of the file was falready persisted, tail_files will not apply. Harvesting will continue at the previous offset. To apply tail_files to all files, you must stop Filebeat and remove the registry file, Be aware that doing this removes ALL previous state
• NOTE : you can use this setting to avoid indexing old log lines when you run Filebeat on a set of log files for the first time. After the first run, we recommend disabling this option, or you risk losing lines during file rotation.