search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

ASCII issue #59

Open paddychang opened 1 year ago

paddychang commented 1 year ago

Hi there,

Please help with the error message below. I run the downloader in Docker. Currently, it throws out error messages again and again and the file 12873_2574.log is getting bigger. The downloader won't move to the next log file 12873_2575 for example.

The version is 2.2.1.

2023-06-02 12:07:55,531 INFO Saving file 12873_2574.log locally to the 'fail' folder 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128) Traceback (most recent call last): File "/usr/local/bin/imperva-connector.py", line 226, in handle_file self.handle_log_decrypted_content(logfile, decrypted_file) File "/usr/local/bin/imperva-connector.py", line 286, in handle_log_decrypted_content local_file.writelines(decrypted_file) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

2023-06-02 12:07:55,540 ERROR Failed to download file 12873_2574.log. Error is - write() argument must be str, not bytes , Traceback (most recent call last): File "/usr/local/bin/imperva-connector.py", line 226, in handle_file self.handle_log_decrypted_content(logfile, decrypted_file) File "/usr/local/bin/imperva-connector.py", line 286, in handle_log_decrypted_content local_file.writelines(decrypted_file) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/imperva-connector.py", line 146, in get_log_files success = self.handle_file(next_file) File "/usr/local/bin/imperva-connector.py", line 237, in handle_file file.write(result[1]) TypeError: write() argument must be str, not bytes

joeymoore commented 1 year ago

Are you compressing and/or encrypting the log from the Management Console? If so, is this configured properly in the script config file? image

paddychang commented 1 year ago

Hi Joe,

Thank you for your reply. We disabled the compress logs and restart the downloader. However, the errors are still coming out. Is there any way we can try to fix it?

Thanks.

paddychang commented 1 year ago

We enable to compress logs again because of the following info from the Imperva website. Note: If you are using the pull mode to download your logs using the API Connector (Python script), compressed files must be used. Uncompressed files will result in an error (-3).

There is a file LastKnownDownloadedFileId.txt to record the current log file. Can I use it to skip the log to the next one?

joeymoore commented 1 year ago

Hello @paddychang, is it possible to use the 3.0.0 release, you should be having these issue in general but the best option would be to use the latest. Cheers