search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

download thread/process stuck when 0 new logs to download. #82

Closed krikui closed 4 months ago

krikui commented 6 months ago

user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ !1990 sudo python3 LogsDownloader.py -c ./path_to_config_folder 2024-01-16 09:57:25,029 MainThread INFO LogsDownloader initializing is done 2024-01-16 09:57:25,029 process_thread INFO Downloading logs index file... 2024-01-16 09:57:26,007 process_thread INFO Downloaded https://logs1.incapsula.com/10057_1791505/logs.index 2024-01-16 09:57:26,007 process_thread INFO 2 new logs to download. 2024-01-16 09:57:26,446 Thread-12 INFO Downloaded https://logs1.incapsula.com/10057_1791505/10057_1433.log 2024-01-16 09:57:26,452 Thread-12 INFO File /home/user/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script/process/10057_1433.log saved successfully 2024-01-16 09:57:26,452 Thread-22 INFO Downloaded 10057_1433.log 2024-01-16 09:57:26,452 Thread-22 INFO Update the complete.log with downloaded file(s). 2024-01-16 09:57:26,955 Thread-14 INFO Downloaded https://logs1.incapsula.com/10057_1791505/10057_1434.log 2024-01-16 09:57:26,956 Thread-14 INFO File /home/user/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script/process/10057_1434.log saved successfully 2024-01-16 09:57:26,956 Thread-22 INFO Downloaded 10057_1434.log 2024-01-16 09:57:26,956 Thread-22 INFO Update the complete.log with downloaded file(s). 2024-01-16 09:57:26,958 process_thread INFO Downloading logs index file... 2024-01-16 09:57:27,361 process_thread INFO Downloaded https://logs1.incapsula.com/10057_1791505/logs.index 2024-01-16 09:57:27,362 process_thread INFO 0 new logs to download. 2024-01-16 09:57:30,366 process_thread INFO Downloading logs index file... 2024-01-16 09:57:30,911 process_thread INFO Downloaded https://logs1.incapsula.com/10057_1791505/logs.index 2024-01-16 09:57:30,912 process_thread INFO 0 new logs to download. 2024-01-16 09:57:33,916 process_thread INFO Downloading logs index file... 2024-01-16 09:57:34,322 process_thread INFO Downloaded https://logs1.incapsula.com/10057_1791505/logs.index 2024-01-16 09:57:34,323 process_thread INFO 0 new logs to download. 2024-01-16 09:57:37,326 process_thread INFO Downloading logs index file... 2024-01-16 09:57:37,599 process_thread INFO Downloaded https://logs1.incapsula.com/10057_1791505/logs.index 2024-01-16 09:57:37,600 process_thread INFO 0 new logs to download. .... ....

2024-01-16 10:09:54,906 process_thread INFO Downloaded https://logs1.incapsula.com/10057_1791505/logs.index 2024-01-16 10:09:54,907 process_thread INFO 0 new logs to download. ^C2024-01-16 10:09:56,102 MainThread INFO Got a termination signal, will now shutdown and exit gracefully 2024-01-16 10:09:56,102 MainThread WARNING Pool Set to False. 2024-01-16 10:09:56,107 MainThread WARNING Pool Terminated. 2024-01-16 10:09:56,110 MainThread WARNING Pool Joined. 2024-01-16 10:09:57,911 process_thread INFO Exiting get_index_file.

^Z [1]+ Stopped sudo python3 LogsDownloader.py -c ./path_to_config_folder user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ ps -aux |grep -i LogsDownloader root 480 0.0 0.0 11028 4588 pts/0 T 09:57 0:00 sudo python3 LogsDownloader.py -c ./path_to_config_folder root 481 0.0 0.4 1662248 30040 pts/0 Tl 09:57 0:00 python3 LogsDownloader.py -c ./path_to_config_folder user 512 0.0 0.0 8168 652 pts/0 S+ 10:11 0:00 grep --color=auto -i LogsDownloader user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ kill -9 480 -bash: kill: (480) - Operation not permitted user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ kill -9 481 -bash: kill: (481) - Operation not permitted user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ ps -aux |grep -i LogsDownloader root 480 0.0 0.0 11028 4588 pts/0 T 09:57 0:00 sudo python3 LogsDownloader.py -c ./path_to_config_folder root 481 0.0 0.4 1662248 30040 pts/0 Tl 09:57 0:00 python3 LogsDownloader.py -c ./path_to_config_folder user 514 0.0 0.0 8168 712 pts/0 S+ 10:12 0:00 grep --color=auto -i LogsDownloader user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ sudo kill -9 481 480 [1]+ Killed sudo python3 LogsDownloader.py -c ./path_to_config_folder user@T480-172905-2:~/proj/incapsula-logs-downloader/incapsula-logs-downloader-master/script$ ps -aux |grep -i LogsDownloader user 518 0.0 0.0 8168 712 pts/0 S+ 10:12 0:00 grep --color=auto -i LogsDownloader

joeymoore commented 6 months ago

@krikui The logs downloader is likelly not stuck, just waiting for more logs. You can download the index by going to https://logs1.incapsula.com/10057_1791505/logs.index and entering your basic authh API creds. If the last index in the log matches the last log downloaded, then we're waiting on new logs.

joeymoore commented 5 months ago

Hello @krikui , does this problem still persist?