search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

zlib.error: Error -3 while decompressing data: incorrect header check #36

Closed qadv closed 2 years ago

qadv commented 2 years ago

TRACELOGS:

2021-10-06 14:00:19,729 INFO Sleeping for 30 seconds before trying to fetch logs again... 2021-10-06 14:00:49,760 INFO No last downloaded file is found - downloading index file and starting to download all the log files in it 2021-10-06 14:00:49,760 INFO Downloading logs index file... 2021-10-06 14:00:50,230 INFO Successfully downloaded file from URL https://logs1.incapsula.com/9264_1873202/logs.index 2021-10-06 14:00:50,231 INFO No last index found, will now scan the entire index... 2021-10-06 14:00:50,584 INFO Successfully downloaded file from URL https://logs1.incapsula.com/9264_1873202/9264_1127.log 2021-10-06 14:00:50,584 INFO Saving file 9264_1127.log locally to the 'fail' folder Error -3 while decompressing data: incorrect header check Traceback (most recent call last): File "incapsula-logs-downloader/script/LogsDownloader.py", line 224, in handle_file decrypted_file = self.decrypt_file(result[1], logfile) File "incapsula-logs-downloader/script/LogsDownloader.py", line 302, in decrypt_file uncompressed_and_decrypted_file_content = zlib.decompressobj().decompress(file_log_content) zlib.error: Error -3 while decompressing data: incorrect header check

2021-10-06 14:00:50,585 ERROR Failed to downloading index file and starting to download all the log files in it - write() argument must be str, not bytes, Traceback (most recent call last): File "incapsula-logs-downloader/script/LogsDownloader.py", line 224, in handle_file decrypted_file = self.decrypt_file(result[1], logfile) File "incapsula-logs-downloader/script/LogsDownloader.py", line 302, in decrypt_file uncompressed_and_decrypted_file_content = zlib.decompressobj().decompress(file_log_content) zlib.error: Error -3 while decompressing data: incorrect header check

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "incapsula-logs-downloader/script/LogsDownloader.py", line 131, in get_log_files self.first_time_scan() File "incapsula-logs-downloader/script/LogsDownloader.py", line 200, in first_time_scan success = self.handle_file(log_file_name) File "incapsula-logs-downloader/script/LogsDownloader.py", line 237, in handle_file file.write(result[1]) TypeError: write() argument must be str, not bytes

(venv) [nkelly@nklablec01 imperva]$ pip3 list Package Version

asn1crypto 0.24.0 Babel 2.5.1 backports.entry-points-selectable 1.1.0 cffi 1.11.5 chardet 3.0.4 cloud-init 20.3 configobj 5.0.6 cryptography 3.2.1 dbus-python 1.2.4 decorator 4.2.1 distlib 0.3.3 ethtool 0.14 filelock 3.3.0 gpg 1.13.1 html5lib 0.999999999 idna 2.5 importlib-metadata 4.8.1 importlib-resources 5.2.2 iniparse 0.4 isc 2.0 Jinja2 2.10.1 jsonpatch 1.21 jsonpointer 1.10 jsonschema 2.6.0 lxml 4.2.3 M2Crypto 0.35.2 MarkupSafe 0.23 netifaces 0.10.6 nftables 0.1 oauthlib 2.1.0 pciutils 2.3.6 perf 0.1 pexpect 4.3.1 pip 21.2.4 platformdirs 2.4.0 ply 3.9 prettytable 0.7.2 psutil 5.4.3 ptyprocess 0.5.2 pycairo 1.16.3 pycparser 2.14 pycrypto 2.6.1 pydbus 0.6.0 PyGObject 3.28.3 pyinotify 0.9.6 PyJWT 1.6.1 pyOpenSSL 19.0.0 pyserial 3.1.1 PySocks 1.6.8 python-dateutil 2.6.1 python-dmidecode 3.12.2 python-linux-procfs 0.6.3 pytz 2017.2 pyudev 0.21.0 PyYAML 3.12 requests 2.20.0 rhnlib 2.8.6 rpm 4.14.3 schedutils 0.6 selinux 2.9 sepolicy 1.1 setools 4.3.0 setroubleshoot 1.1 setuptools 39.2.0 six 1.11.0 slip 0.6.4 slip.dbus 0.6.4 sos 4.0 SSSDConfig 2.4.0 subscription-manager 1.28.13 syspurpose 1.28.13 systemd-python 234 typing-extensions 3.10.0.2 urllib3 1.24.2 virtualenv 20.8.1 webencodings 0.5.1 wheel 0.37.0 zipp 3.6.0

(venv) [nkelly@nklablec01 imperva]$ cat /etc/system-release CentOS Linux release 8.4.2105 (venv) [nkelly@nklablec01 imperva]$ python3 --version Python 3.6.8

Compression is not configured nor is encryption.

qadv commented 2 years ago

https://github.com/imperva/incapsula-logs-downloader/blob/8dae16137c78bfc75b213853ccfc6553f09f8e70/script/LogsDownloader.py#L302

Has no logic to determine if the file is compressed or not (Console offers the ability to not compress files) - will see if I can come up with some basic log but for now have just removed the compression processing from the line.

JCoffey85 commented 2 years ago

https://github.com/imperva/incapsula-logs-downloader/blob/8dae16137c78bfc75b213853ccfc6553f09f8e70/script/LogsDownloader.py#L302

Has no logic to determine if the file is compressed or not (Console offers the ability to not compress files) - will see if I can come up with some basic log but for now have just removed the compression processing from the line.

same issue. @qadv what was your work around?. I also have a ticket with support opened for this

mveroone commented 2 years ago

on line ~302 of the script : replace this :

            uncompressed_and_decrypted_file_content = zlib.decompressobj().decompress(file_log_content)

by

            uncompressed_and_decrypted_file_content = file_log_content

Of course it would be better for the script to autodetect whether the data is compressed or not. Or have the information in the file header.

qadv commented 2 years ago

Definitely!

Regards, Noel Kelly M: +61 481 830 450

On 6 Nov 2021, at 12:13 am, Maxime Véroone @.***> wrote:



on line ~302 of the script : replace this :

        uncompressed_and_decrypted_file_content = zlib.decompressobj().decompress(file_log_content)

by

        uncompressed_and_decrypted_file_content = file_log_content

Of course it would be better for the script to autodetect whether the data is compressed or not. Or have the information in the file header.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/36#issuecomment-961929280, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4JQSISGYLF4TMMT5G7XQLUKPRB7ANCNFSM5FNMT3QQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

JCoffey85 commented 2 years ago

Had a support session with Imperva. they stated my error was due to clicking compressed then uncompressed in the integration settings. I believe that once the index file / log file that failed to decompress expired, and the LastKnownDownloaded.txt removed, I was able to start ingesting again w/o error. been running about 2 weeks now w/o problems. I am running compressed.

qadv commented 2 years ago

Congratulations, the responsibility being on you to change your configuration is however not true – the logic in the python doesn’t account for the change in compression settings during the lifecycle of the configuration / nor does the configuration limit your ability to change between compressed and uncompressed.

Yes, you can arbitrarily make the configuration align (in fact you can change it to support a period of compressed vs. not compressed files and make this work); all this is moot though and the supported log collection method does not cater for the configuration options available in terms of log export.

I don’t have time to fix (nor am I the greatest python coder) but it feels very much like a rookie ‘in 5 lines or less’ problem to solve given the use of standard libraries.

NJK

From: JCoffey85 @.> Date: Tuesday, 9 November 2021 at 11:50 pm To: imperva/incapsula-logs-downloader @.> Cc: Noel Kelly @.>, Mention @.> Subject: Re: [imperva/incapsula-logs-downloader] zlib.error: Error -3 while decompressing data: incorrect header check (#36)

Had a support session with Imperva. they stated my error was due to clicking compressed then uncompressed in the integration settings. I believe that once the index file / log file that failed to decompress expired, and the LastKnownDownloaded.txt removed, I was able to start ingesting again w/o error. been running about 2 weeks now w/o problems. I am running compressed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/36#issuecomment-964169100, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4JQSJ6ZDI4PYAQC6VVDALULERLDANCNFSM5FNMT3QQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

cschmidt0121 commented 2 years ago

This should handle both cases (compress vs uncompressed) so you don't have to mess around with integration settings

https://github.com/imperva/incapsula-logs-downloader/pull/40

charm-lor commented 2 years ago

To troubleshoot, I placed the following code on line 300 before it checks for the file_encryption_key to see if the file was encrypted or not. The log was downloaded was indeed encrypted. However, by adding that code, somehow the script ran successfully. I've commented out the code and the script continues to run successfully. Weird.

    tmp_file = open("/tmp/" + filename,"wb")
    tmp_file.write(file_log_content)
    tmp_file.close()