search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

2 issues causing the script to hangup and never move on #13

Closed connor-borchgrevink closed 8 months ago

connor-borchgrevink commented 4 years ago

Hello!

I am running into two separate issues that both causing the script to hang up and stop downloading the logs.

  1. The first issue is if the log has a non-ASCII character, the script hangs up on trying to write the line and never moves past it. I added a try/except at line... 275 if self.config.SAVE_LOCALLY == "YES": local_file = open(self.config.PROCESS_DIR + filename, "a+") try: local_file.writelines(decrypted_file) except: pass

It seems to have worked? I haven't had the same error message since adding it.

  1. If the next file id can't be found the script continuously checks for that id and never moves past it. The only way I seem to be able to break the loop is by deleting LastKnownDownloadedFileID.txt and recreating it. It is not a sustainable practice. Thoughts?

2020-06-23 09:32:38,161 INFO Sleeping for 5 seconds until next file download retry number 1 out of 3 2020-06-23 09:32:43,540 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:32:43,542 INFO Sleeping for 5 seconds until next file download retry number 2 out of 3 2020-06-23 09:32:48,943 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:32:48,944 INFO Sleeping for 5 seconds until next file download retry number 3 out of 3 2020-06-23 09:32:54,286 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:32:54,287 INFO Could not get log file 1982_2902779.log. It could be that the log file does not exist yet. 2020-06-23 09:32:54,287 INFO Sleeping for 30 seconds before trying to fetch logs again... 2020-06-23 09:33:24,648 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:33:24,649 INFO Sleeping for 5 seconds until next file download retry number 1 out of 3 2020-06-23 09:33:30,026 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:33:30,027 INFO Sleeping for 5 seconds until next file download retry number 2 out of 3 2020-06-23 09:33:35,311 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:33:35,312 INFO Sleeping for 5 seconds until next file download retry number 3 out of 3 2020-06-23 09:33:40,672 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:33:40,673 INFO Could not get log file 1982_2902779.log. It could be that the log file does not exist yet. 2020-06-23 09:33:40,673 INFO Sleeping for 30 seconds before trying to fetch logs again... 2020-06-23 09:34:11,044 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:34:11,045 INFO Sleeping for 5 seconds until next file download retry number 1 out of 3 2020-06-23 09:34:16,867 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:34:16,868 INFO Sleeping for 5 seconds until next file download retry number 2 out of 3 2020-06-23 09:34:22,153 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:34:22,154 INFO Sleeping for 5 seconds until next file download retry number 3 out of 3 2020-06-23 09:34:27,497 WARNING Could not find file https://logs1.incapsula.com/1982_2438/1982_2902779.log. Response code is 404 2020-06-23 09:34:27,499 INFO Could not get log file 1982_2902779.log. It could be that the log file does not exist yet. 2020-06-23 09:34:27,499 INFO Sleeping for 30 seconds before trying to fetch logs again...

Thumbscrew commented 3 years ago

For the first issue I changed the following line:

local_file = open(self.config.PROCESS_DIR + filename, "a+")

To:

local_file = open(self.config.PROCESS_DIR + filename, "a+", encoding='utf-8')

Without specifying the encoding for the output file it was throwing an exception (it was trying to use 'cp1252' encoding which doesn't like non-ASCII chars).

tsunaryo commented 3 years ago

anyone find a solution for the second issue?

mveroone commented 2 years ago

Having the same here. even if the script runs, it seems to sometimes fail to catch up with how fast new logs appear and since it's waiting 5seconds between retries, it losses ground...

But it also seems incapsula is not providing any support for this script so time to dig into the code and try to fix ourselves.

AVitg commented 2 years ago

@mveroone changed some timers, check #42

joeymoore commented 1 year ago

Here please let me know if this is still an issue. A new 3.0.0 release is available and resolves a lot of known issues. Please review the readme if planning to upgrade. https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta

secxsast commented 1 year ago

Can someone help me I'm getting the same Warning "WARNING Could not find file https://logs1.incapsula.com/"

Has anyone here managed to solve it?

joeymoore commented 1 year ago

Do you have the full URI in the configuration? With you specific information like https://logs1.incapsula.com/1982_2438/1982_2902779.log? Are you using version 3.0.0?