Incorrect handling of UTF-8 results in decompressed log content file write failures

[clp71s]

In LogsDownloader.py lines 284 through 286, decompressed log entries are written to an output file.

    if self.config.SAVE_LOCALLY == "YES":
        local_file = open(self.config.PROCESS_DIR + filename, "a+")
        local_file.writelines(decrypted_file)

Occasionally, logs would fail to write (and were instead saved in compressed format to the "fail" directory.

Review of the output content (when directly printed to stdout) did not identify any issue with the content itself, however using type() on the decrypted_file object indicated it to be unicode. It appears that unicode can occasionally interfere with writelines causing a type error "TypeError: writelines() argument must be a sequence of strings".

The error appears to be resolved (subject to further testing) by implementing the following change:

    if self.config.SAVE_LOCALLY == "YES":
        local_file = open(self.config.PROCESS_DIR + filename, "a+")
        local_file.writelines(decrypted_file.encode('utf-8'))

Can anyone else confirm they have encountered this sort of issue and if this suggested change successfully resolves it without introducing additional errors?

[delink]

@clp71s I came across this issue also and tried your suggestion, but we're running this under Python3, and we were getting an additional error:

2022-01-28 11:00:03,032 ERROR Failed to download file 6132_206629.log. Error is - write() argument must be str, not bytes , Traceback (most recent call last):
  File "/opt/incapsula_cloud_waf_downloader/bin/LogsDownloader.py", line 226, in handle_file
    self.handle_log_decrypted_content(logfile, decrypted_file)
  File "/opt/incapsula_cloud_waf_downloader/bin/LogsDownloader.py", line 286, in handle_log_decrypted_content
    local_file.writelines(str(decrypted_file.encode('utf-8')))
TypeError: write() argument must be str, not int

Given that it just seemed to need the bytes cast back into a string, I made a small modification to your fix to do so, and it solved the issue on our side.

--- LogsDownloader.py.orig      2021-11-23 10:02:48.271100102 -0600
+++ LogsDownloader.py   2022-01-28 10:59:42.354200133 -0600
@@ -283,7 +283,7 @@

         if self.config.SAVE_LOCALLY == "YES":
             local_file = open(self.config.PROCESS_DIR + filename, "a+")
-            local_file.writelines(decrypted_file)
+            local_file.writelines(str(decrypted_file.encode('utf-8')))

     """
     Decrypt a file content

This fixed the problem for us running on Ubuntu 20.04 under Python3.

[NySok888]

@delink

I followed yours but still got same error.

############## error log##### File "/etc/incapsula/logs/script/LogsDownloader.py", line 222, in handle_file self.handle_log_decrypted_content(logfile, decrypted_file) File "/etc/incapsula/logs/script/LogsDownloader.py", line 255, in handle_log_decrypted_content decrypted_file = decrypted_file.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte.

############################ Confgure ############## if self.config.SAVE_LOCALLY == "YES": local_file = open(self.config.PROCESS_DIR + filename, "a+") local_file.writelines(str(decrypted_file.encode('utf-8'))) """ Decrypt a file content

@delink any advise from you?

[joeymoore]

@NySok888 @clp71s @delink please let me know if this is still an issue. A new 3.0.0 release is available and resolves a lot of known issues. Please review the readme if planning to upgrade. https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta