imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

Log decryption Error and saved to fail folder #19

Closed zhinobu38 closed 3 years ago

zhinobu38 commented 3 years ago

Hi,

I'm running on Windows Server 2012R2 with Python 3.8.6 version. Some log file cause below error at decryption state. Could you please suggest on how to fix this issue?

Thank you.

2020-11-04 15:00:07,760 INFO Successfully downloaded file from URL https://logs1.incapsula.com/6827_1XXX7/6827_21977.log 2020-11-04 15:00:07,760 WARNING Keys Dir: C:\incapsula\logs-downloader-master2-Test\config\keys 2020-11-04 15:00:07,776 INFO Syslog enabled, using UDP 2020-11-04 15:00:14,870 INFO Saving file 6827_21977.log locally to the 'fail' folder 'charmap' codec can't encode character '\u014c' in position 0: character maps to Traceback (most recent call last): File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 222, in handle_file self.handle_log_decrypted_content(logfile, decrypted_file) File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 276, in handle_log_decrypted_content local_file.writelines(decrypted_file) File "C:\Python38\lib\encodings\cp874.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u014c' in position 0: character maps to

zhinobu38 commented 3 years ago

As I searchIt seems to be window's terminal issue that not support some encoded character

So I try to change the script, line 255 as below, to avoid this issue. from : decrypted_file = decrypted_file.decode('utf-8') to : decrypted_file = decrypted_file.decode('utf-8', 'ignore')

But it still running the same error. Please help suggest.

Thank you so much in advanced.

the log which i test today is below : 2020-11-06 11:01:49,027 INFO Sleeping for 2 seconds before fetching the next logs file 2020-11-06 11:01:57,216 INFO Successfully downloaded file from URL https://logs1.incapsula.com/6827_XXXXX/6827_26545.log 2020-11-06 11:01:57,216 WARNING Keys Dir: C:\incapsula\logs-downloader-master2-Test\config\keys 2020-11-06 11:01:57,232 INFO Syslog enabled, using UDP 2020-11-06 11:01:58,966 INFO Saving file 6827_26545.log locally to the 'fail' folder 'charmap' codec can't encode character '\xe3' in position 0: character maps to Traceback (most recent call last): File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 222, in handle_file self.handle_log_decrypted_content(logfile, decrypted_file) File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 277, in handle_log_decrypted_content local_file.writelines(decrypted_file) File "C:\Python38\lib\encodings\cp874.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\xe3' in position 0: character maps to

2020-11-06 11:01:58,966 ERROR Failed to download file 6827_26545.log. Error is - write() argument must be str, not bytes , Traceback (most recent call last): File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 222, in handle_file self.handle_log_decrypted_content(logfile, decrypted_file) File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 277, in handle_log_decrypted_content local_file.writelines(decrypted_file) File "C:\Python38\lib\encodings\cp874.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\xe3' in position 0: character maps to

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 143, in get_log_files success = self.handle_file(next_file) File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 233, in handle_file file.write(result[1]) TypeError: write() argument must be str, not bytes

2020-11-06 11:02:46,998 INFO Successfully downloaded file from URL https://logs1.incapsula.com/6827_XXXXX/6827_26545.log

zhinobu38 commented 3 years ago

I found solution from Imperva Communities. The issue is about encoding issue with Windows system running in Asia locale that lead to unsupported encoding character.

To fix this replace inline 275 on Logdownloader.py with : local_file = open(self.config.PROCESS_DIR + filename, "a+", encoding="UTF-8")

This address my problem.