Amazon S3 input - Githubissues

sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint

https://sematext.com/logagent

Apache License 2.0

389 stars 79 forks source link

Amazon S3 input #91

Open megastef opened 7 years ago

megastef commented 7 years ago

Logagent should have a plugin 1) to read S3 data as soon new files are created 2) to import historical data

prog8 commented 7 years ago

For initial dump Logagent probably needs to ls a bucket and read all files
For new data Logagent probably has to subscribe to SNS

otisg commented 7 years ago

... or simply check periodically?

prog8 commented 7 years ago

I think a periodical check is no-go. A bucket can have thousands, tens thousands or hundreds of thousands of objects and ls will not be fast enough.

megastef commented 7 years ago

Perhaps we just need an example configuration and documentation to combine s3cmd with other Logagent plugins:

Logagent command plugin to start S3-Sync http://s3tools.org/s3cmd-sync s3cmd sync --exclude '*.zip' --include 'dir2/*.log' . s3://s3tools-demo/demo/
Logagent file input plugin to read downloaded files
Logagent Elasticsearch output plugin to stream files to Sematext Cloud or Elasticsearch ...

megastef commented 7 years ago

Documentation links for plugins: https://sematext.com/docs/logagent/#plugins

megastef commented 6 years ago

Helpful S3 download module: https://www.npmjs.com/package/s3-download-stream