search

cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

[Feature: multithreading]Slow Pull Time #3

Closed danielkasen closed 6 years ago

danielkasen commented 6 years ago

So using this against something like ELB logs it's having difficulty keeping up. I see that it runs in a single threaded operation. So correct me if I'm wrong, but it grabs a single queue event, then downloads and process before moving on? Is there any way to increase the number of threads on this input module so I can say, perform 10 downloads at once? I have no back pressure from my filters and output at the moment, but my sqs queue only grows.

cherweg commented 6 years ago

Hey Daniel, multitheading is on my feature list. At this time you could duplicate you input configuration to get more download workers.

regards Christian

danielkasen commented 6 years ago

oh that's genius! Just add another input block. Thanks for the tip. :)

danielkasen commented 6 years ago

Hmm sadly that actually decreased my total throughput, I liked the idea though. Guess each input thread is probably conflicting with each other since they're on the same default delay.

cherweg commented 6 years ago

Hey Daniel, I‘m really sorry about that. I‘ll try to release an early version of a multithread implementation within the next weeks.

Feel free to make a pull request.

Regards Christian

cherweg commented 6 years ago

@danielkasen: Big changes in 1.4.0: Multithreading and new File handling.

Please try if it fixes your problem ...

Christian

danielkasen commented 6 years ago

Woot, excited to try it out tomorrow!

On Mar 17, 2018 12:11 AM, "cherweg" notifications@github.com wrote:

@danielkasen https://github.com/danielkasen: Big changes in 1.4.0: Multithreading and new File handling.

Please try if it fixes your problem ...

Christian

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cherweg/logstash-input-s3-sns-sqs/issues/3#issuecomment-373763680, or mute the thread https://github.com/notifications/unsubscribe-auth/AeFp1OyIUT_V9Eg1dxyXKSPDEOKlvTl7ks5te-RJgaJpZM4SXzcS .

-- *Grab is hiring. Learn more at *https://grab.careers https://grab.careers/

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email and notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

cherweg commented 6 years ago

Tested the new implementation yesterday. Needs less heap and ist faster on big files. Great performance improofment in my special case.

Performance is good enough i think ... Next we will implement some aws auth features.

christian