srebhan
commented
4 days ago @IvanoCar please test the binary in PR #15569, available once CI finished the tests, and let me know if that works for you. You should be able to start from this config
# Send telegraf metrics to file(s) in a remote filesystem
[[outputs.remotefile]]
## Remote location according to https://rclone.org/#providers
## Check the backend configuration options and specify them in
## <backend type>[,<param1>=<value1>[,...,<paramN>=<valueN>]]:[root]
## for example:
remote = 's3,provider=AWS,access_key_id=your-access-key,secret_access_key=your-secret-key,session_token=your-token,region=eu-north-1:mybucket'
## Files to write in the remote location
## Each file can be a Golang template for generating the filename from metrics.
## See https://pkg.go.dev/text/template) for a reference and use the metric
## name (`{{.Name}}`), tag values (`{{.Tag "name"}}`), field values
## (`{{.Field "name"}}`) or the metric time (`{{.Time}}) to derive the
## filename.
files = ['{{.Name}}-{{.Time.Format "2006-01-02"}}']
## Use batch serialization format instead of line based delimiting.
## The batch format allows for the production of non-line-based output formats
## and may more efficiently encode metrics.
# use_batch_format = false
## Cache settings
## Time to wait for all writes to complete on shutdown of the plugin.
# final_write_timeout = "10s"
## Time to wait between writing to a file and uploading to the remote location
# cache_write_back = "5s"
## Maximum size of the cache on disk (infinite by default)
# cache_max_size = -1
## Data format to output.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
data_format = "influx"
IvanoCar
Use Case
I would like to open a pull request so I get input from the community. The use case here is output metrics from inputs and then output the metrics on a S3 bucket on a specific path.
Having data ingested via Telegraf on S3 which is used, for example, as a datalake is useful because it can be used in various analytics purposes and can be considered as enrichment of data already available from various other sources.
Expected behavior
I expect files to be written to S3 bucket and specified subfolders. Auth can be handled via IAM user on AWS.
Actual behavior
This currently is not supported in Telegraf.
Additional info
Config could look like this: