search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.39k stars 1.51k forks source link

New source: lambda runtime logs #5326

Open ctaintor opened 3 years ago

ctaintor commented 3 years ago

Use-cases

Part of the value of Vector that I see is removing unnecessary components from the log stream. AWS recently launched realtime lambda logs as an extension and this can remove the need for things like Kinesis Firehose.

Attempted Solutions

Today the only solution is using Kinesis Firehose for getting logs out of Lambda.

Proposal

Add a source for lambda runtime logs to make it simpler for people to get their logs into Vector

References

binarylogic commented 3 years ago

Great idea! Thanks for opening.

andreyvital commented 3 years ago

@binarylogic any updates on that? 🤔

wperron commented 2 years ago

I was actually looking for something like that, I'm already using Vector as an agent on my EC2 instances, but now I have some Lambda functions that I'd like to be able to control where their logs are being sent. I've got some free time on my hand for the next couple of days, I can start looking into it.

Phizicks commented 2 years ago

@wperron how did you go with looking into making vector work as a lambda extension? I've worked on CW subscription to kinesis where vector picks it up and cross account kinesis has been a pain. I much prefer the lambda extension which allows me to add my own meta-data/tagging where CW doesn't.

wperron commented 2 years ago

Honestly I haven't had time to look into it much. I started investigating a bit but then life happened and completely forgot about this. I might take another look though, it's an interesting issue.

Phizicks commented 2 years ago

damn life, always gets in the way. I almost divorced mine but doctor said it was a bad idea.

jszwedko commented 2 years ago

Noting a user did do a POC of this:

I'm working on a POC with Vector running it within AWS Lambda. I managed to run Vector as an extension using theAWS Lambda Extension API.

https://github.com/vectordotdev/vector/issues/12833

chriszs commented 1 year ago

Seems like Lambda's new Telemetry API might make this more effective.

RichiCoder1 commented 1 year ago

Yup, I'd amend this issue to use the Telemetry API.

Especially interested in this as this issue plus #1444 could hopefully the replace the less performant otel layer provided by AWS.

sandys commented 1 year ago

hi folks, any chance there is a poc for this ? there is an example for doing this here https://github.com/megaproaktiv/Lambda-Telemetry-API-Loki/tree/main/loki_extension