search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

speedup file handling, for busy environment #42

Closed AVitg closed 9 months ago

AVitg commented 2 years ago

In busy environments, the sleep/wait timers are way to big

joeymoore commented 1 year ago

@AVitg would you agree that we can close this with the release of 3.0.0? If not, it looks like a simple merge.

AVitg commented 1 year ago

@joeymoore, let's say... The biggest issue with the pre v3.0.0 version is to get back on track, when the "logid" is not there(anymore or so).

Tbh I would consider the "old" version kind of broken for this functionality anyhow I'm changing our environment to v3; so its up to you, to merge, or not.

Btw. Thanks for v3!

joeymoore commented 1 year ago

This is great point @AVitg , I will look at a way to "start" from where the old script left of on the download process and communicate with my coworker on getting this merge. @AaronSeibert

AVitg commented 1 year ago

I also tried to "repair" the old( current main branch) in a way, that it continues to iterate the counter... I.e. the issue I referenced in here from the other person. I never tried it in our production, as I do think the v3 script solves the underlying issue way more elegant, hence no other PR for this. If your interested happy to share, however I am not sure about how reliable it does it's job. Hence I changed to v3.