search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
141 stars 275 forks source link

[Cloudfront S3 Logs for production project website into AOS Domain] #4658

Open nateynateynate opened 7 months ago

nateynateynate commented 7 months ago

Is your feature request related to a problem? Please describe

I'd love to be able to dashboard up some of our website logs. I don't really have a 'permanent' solution other than adding temporary creds to my environment, pulling all the gzips from the S3 bucket, unzipping them, and then data-preppering them into the OS instance. I need something robust and long standing.

Describe the solution you'd like

If we could trigger a lambda when the S3 bucket receives a new file that indexes the lines into an AWS OpenSearch domain or even just a monolithic EC2 instance with OS installed, that would be awesome.

Describe alternatives you've considered

I have a ReadOnlyRole, but I don't have a programmatic way of assuming that role, downloading any new files in the S3 bucket, and then indexing them by hand by releasing Data Prepper on them.

Additional context

We have no current means of doing this, and I'd sure love to define some health metrics regarding site visits, referrers, etc. that doesn't involve any external things like Google Analytics.

jordarlu commented 7 months ago

Thanks for the posting of requesting a new feature, @nateynateynate , while I am not certain if this is the best repo for your request, I am adding a few folks to comment.. CC: @peterzhuamazon @gaiksaya

prudhvigodithi commented 6 months ago

[Triage] Thanks @jordarlu and @nateynateynate, pushing logs to OpenSearch and visualizing them through Dashboard is a good idea, but may I know what specific logs are referring here? I have seen you mentioned website logs, can you add more context to this? Thanks @bbarani @zelinh