[inputs.tail] Logs being processed everytime the agent restarts

[bloodmc]

Relevant telegraf.conf

[[inputs.tail]]
  files = ["\\\\servername\\logs\\powershell\\**\\*.txt"]
  from_beginning = true
  watch_method = "poll"
  data_format = "grok"
  ## Define custom Grok patterns for matching the PowerShell log structure
  grok_custom_patterns = '''
    BASE64_ENCODED (?i)[A-Za-z0-9+/]{44,}(?:[A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)
    SUSPICIOUS_COMMANDS (?i)reflection|socket|download|internetexplorer.application|xmlhttp|assemblybuilder|gzipstream|decompress|io.compression|write-zip|(expand|compress)-archive|-bxor|security.cryptography|getdelegateforfunctionpointer
    POWERSHELL_USAGE (?i)powershell -version|invoke-command|invoke-expression|start-process|set-executionpolicy
    NETWORK_ACTIVITY (?i)socket|webclient|wget|curl|net.webclient|downloadstring|downloadfile|uploadfile
    ENCODING_METHODS (?i)frombase64string|base64|utf8|unicode|encode|decode|compress|expand
    MALICIOUS_TOOLS (?i)mimikatz|nishang|metasploit|shellcode|exploit|amsibypass
    SUSPICIOUS_BEHAVIOR (?i)disable-realtime|bypass|enable-psremoting|brute.*force|port.*scan|reverse.*shell|credential.*dump
  '''

  grok_patterns = [
    "%{BASE64_ENCODED:value}",
    "%{SUSPICIOUS_COMMANDS:value}",
    "%{POWERSHELL_USAGE:value}",
    "%{NETWORK_ACTIVITY:value}",
    "%{ENCODING_METHODS:value}",
    "%{MALICIOUS_TOOLS:value}",
    "%{SUSPICIOUS_BEHAVIOR:value}"
  ]

Logs from Telegraf

N/A

System info

Telegraf 1.30.1, Windows Server 2022 Standard and pushing logs to loki

Docker

Manual setup. (No docker used)

Steps to reproduce

1. Start telegraf with above config.
2. All logs will be tailed in server share and matches will be sent to loki
3. Restart the agent and it resends the same log data.

Expected behavior

For restarts to not resend the same log data

Actual behavior

Restarting the agent resends the same log data.

Additional info

No response

[powersj]

Restart the agent and it resends the same log data.

Because you told it to ;)

from_beginning = true

This means to read from the beginning of a file you defined. Please also look at the statefile agent option, which can save the state of certain plugins, like tail, so that when you restart, after a successful shutdown, Telegraf is able to pick back up where it left off.

[bloodmc]

Ah missed that. Thanks!