l-natali commented 2 years ago

I have run script and saw that logs was writting to the selected directory. But after script was stopped and I launched it again I see error: 2022-07-07 12:40:19,749 WARNING Could not find file https://logs1.incapsula.com/xxxxx_xxxxxxx/11149_205.log. Response code is 404 2022-07-07 12:40:19,750 INFO Sleeping for 1 seconds until next file download retry number 2 out of 3

I decreased sleep/wait timer but it didn't help, error still here.

Does anybody know how to fix it?

Thanks in advance!

l-natali commented 2 years ago

Fixed it via LastKnownDownloadedFileId.txt cleaning

joeymoore commented 2 years ago

@l-natali This "error" of 404 is normal as the assumed next log to downloaded may not have been created yet on the platform. Deleting the LastKnown will only cause the script to start downloading logs again from the beginning.

jclo1 commented 1 year ago

Hi @l-natali I have the same error with the script. (404)

did you have any issues removing the LastKnownDownloadedFileId.txt entry?

How far you started collecting the logs again?

n1majne3 commented 1 year ago

@l-natali This "error" of 404 is normal as the assumed next log to downloaded may not have been created yet on the platform. Deleting the LastKnown will only cause the script to start downloading logs again from the beginning.

hi, @joeymoore I found the 10 times retries will easily to reach due to the 404 issue, and onece it "attemptting to revover", it will start to downloading logs again from the beginning. not sure if incresing the the retries could fix?

l-natali commented 1 year ago

Hi @l-natali I have the same error with the script. (404)

did you have any issues removing the LastKnownDownloadedFileId.txt entry?

How far you started collecting the logs again?

Hi @jclo1 I didn't remove LastKnownDownloadedFileId.txt entirely. I cleaned it via vi(m) text editor - open file in vi(m), type "dd" and saved this file. After it straight away I sterted collecting logs again.

mahegs commented 1 year ago

I have encountered this same issue. I have cleaned the LastKnownDownloadedFileId.txt and logs started over, but when they caught up, I again hit the same file that could not be found, and cannot progress beyond that point. I then set the log number in my LastKnownDownloadedFileId.tx to the log right before the one that cannot be found, the one before was successfully downloaded and the next in sequence is not available/ready. Essentially, I have no new log files ready in the last 4 days.

2022-10-19 12:11:58,151 INFO LogsDownloader initializing is done Oct 19 12:11:58 is103.fb bash[16313]: 2022-10-19 12:11:58,975 INFO Successfully downloaded file from URL https://logs1.incapsula.com/xxxx_xxxxxxx/xxxx_6714641.log Oct 19 12:11:59 is103.fb bash[16313]: 2022-10-19 12:11:59,001 INFO File 3094_6714641.log download and processing completed successfully Oct 19 12:11:59 is103.fb bash[16313]: 2022-10-19 12:11:59,001 INFO Sleeping for 2 seconds before fetching the next logs file Oct 19 12:12:01 is103.fb bash[16313]: 2022-10-19 12:12:01,903 INFO Successfully downloaded file from URL https://logs1.incapsula.com/xxxx_xxxxxxx/xxxx_6714642.log Oct 19 12:12:02 is103.fb bash[16313]: 2022-10-19 12:12:02,401 INFO File 3094_6714642.log download and processing completed successfully Oct 19 12:12:02 is103.fb bash[16313]: 2022-10-19 12:12:02,403 INFO Sleeping for 2 seconds before fetching the next logs file Oct 19 12:12:05 is103.fb bash[16313]: 2022-10-19 12:12:05,026 WARNING Could not find file https://logs1.incapsula.com/xxxx_xxxxxxx/xxxx_6714643.log. Response code is 404 Oct 19 12:12:05 is103.fb bash[16313]: 2022-10-19 12:12:05,026 INFO Sleeping for 5 seconds until next file download retry number 1 out of 3

Are there any other steps I can try to get to new data?

mahegs commented 1 year ago

LastKnownDownloadedFileId.tx

I was able to view the sequence of logs available for download via the logs.index. We were indeed missing several in the sequence, which was causing the file not found errors. I was able to update the LastKnownDownloadedFileId.txt to the log before they were complete again. I restarted the service, and logs are downloading again.

These are the logs we have in order, after the missing file: xxxx_6714639.log xxxx_6714641.log xxxx_6714642.log - no 6714643 or 6714644 xxxx_6714645.log - no 6714646 xxxx_6714647.log xxxx_6714648.log xxxx_6714649.log

I updated the LastKnownDownloadedFileId.txt file to list the 671646 file as the last known, and restarted the incapsulaLogs.service and log collection has resumed as expected. I do not know what causes the missing log files, but I have reached out to support for more info.

joeymoore commented 1 year ago

@l-natali @jclo1 @n1majne3 @mahegs please let me know if this is still an issue. A new 3.0.0 release is available and resolves a lot of known issues. Please review the readme if planning to upgrade. https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta

l-natali commented 1 year ago

@l-natali @jclo1 @n1majne3 @mahegs please let me know if this is still an issue. A new 3.0.0 release is available and resolves a lot of known issues. Please review the readme if planning to upgrade. https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta

It doesn't an issue anymore.

mahegs commented 1 year ago

Hi Joe,

I attempted to implement release 3.0.0; however, I am receiving the following message when I execute the script:

python3 LogsDownloader.py

Could Not find Configuration file

I have Python3.7.11 installed and successfully installed the dependencies in requirements.txt. Do you have any suggestions on how to address the error I am receiving?

Thanks! Michal @.***)

From: Joe Moore @.> Sent: Wednesday, March 22, 2023 9:46 AM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@l-natalihttps://urldefense.com/v3/__https:/github.com/l-natali__;!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUjtfjlcf$ @jclo1https://urldefense.com/v3/__https:/github.com/jclo1__;!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUtAUh1LU$ @n1majne3https://urldefense.com/v3/__https:/github.com/n1majne3__;!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUmgKvHJ-$ @mahegshttps://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUj5gsVQb$ please let me know if this is still an issue. A new 3.0.0 release is available and resolves a lot of known issues. Please review the readme if planning to upgrade. https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta__;!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUidcMFgC$

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1479812756__;Iw!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUhXhKAEl$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJAANZ5VU6AZR3NBFS3W5MNDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!CAfOCryhSMLr1Y0mX240lbMWL0xWQn0u_TUuqruLAgF_hh5usEyMNw9Tom2Kj7mAXRquNEe4hNK4ieFP1KOBi0bAUmSy7u6y$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

joeymoore commented 1 year ago

@mahegs did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

mahegs commented 1 year ago

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

From: Joe Moore @.> Sent: Tuesday, April 4, 2023 3:12 PM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@mahegshttps://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$ did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

joeymoore commented 1 year ago

Can you sure your config area for the SYSLOG info? Cheers, Joe

From: mahegs @.> Sent: Wednesday, April 5, 2023 10:08 AM To: imperva/incapsula-logs-downloader @.> Cc: Joe Moore @.>; State change @.> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

From: Joe Moore @.> Sent: Tuesday, April 4, 2023 3:12 PM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@mahegshttps://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$ did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497841731, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA. You are receiving this because you modified the open/close state.Message ID: @.***>

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

mahegs commented 1 year ago

Our previous Settings.Config:

APIID=XXXXXX APIKEY=XXXXXX SAVE_LOCALLY=NO PROCESS_DIR=/tmp/processed/ BASEURL=https://logs1.incapsula.com/3094_1202536/ USEPROXY=YES PROXYSERVER=http://proxy.fb:8080 SYSLOG_ENABLE=YES SYSLOG_ADDRESS=10.0.254.73 SYSLOG_PORT=514 USE_CUSTOM_CA_FILE=NO CUSTOM_CA_FILE= SYSLOG_PROTO=

And this is the updated version. Of note, we had a SAVE_LOCALLY=NO stanza in our previous config that I did carry over, but I ran it with and without it without a noticeable difference.

[SETTINGS] IMPERVA_API_ID=XXXXX IMPERVA_API_KEY=XXXXXX IMPERVA_API_URL=https://logs1.incapsula.com/3094_1202536/ IMPERVA_INCOMING_DIR= IMPERVA_PROCESS_DIR=/tmp/processed/ IMPERVA_ARCHIVE_DIR= IMPERVA_USE_PROXY=YES IMPERVA_PROXY_SERVER=http://proxy.fb:8080 IMPERVA_USE_CUSTOM_CA_FILE=NO IMPERVA_CUSTOM_CA_FILE= IMPERVA_SYSLOG_ENABLE=YES IMPERVA_SYSLOG_ADDRESS=10.0.254.73 IMPERVA_SYSLOG_PORT=514 IMPERVA_SYSLOG_PROTO= IMPERVA_SPLUNK_HEC= IMPERVA_SPLUNK_HEC_IP= IMPERVA_SPLUNK_HEC_PORT= IMPERVA_SPLUNK_HEC_TOKEN= IMPERVA_SPLUNK_HEC_SRC_HOSTNAME= IMPERVA_SPLUNK_HEC_INDEX= IMPERVA_SPLUNK_HEC_SOURCE= IMPERVA_SPLUNK_HEC_SOURCETYPE=

Thanks! Michal

From: Joe Moore @.> Sent: Wednesday, April 5, 2023 11:15 AM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Can you sure your config area for the SYSLOG info? Cheers, Joe

From: mahegs @.<mailto:@.>> Sent: Wednesday, April 5, 2023 10:08 AM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Joe Moore @.<mailto:@.>>; State change @.<mailto:@.>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

From: Joe Moore @.<mailto:@.>> Sent: Tuesday, April 4, 2023 3:12 PM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Hegstrom, Michal @.<mailto:@.>>; Mention @.<mailto:@.>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@mahegs<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E. You are receiving this because you were mentioned.Message ID: @.**@.https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E%20did%20you%20make%20the%20needed%20changes%20to%20the%20config%20file%20as%20all%20the%20variable%20names%20were%20changed%20to%20prepend%20the%20existing%20vars%20with%20IMPERVA_*%0b%0b-%0bReply%20to%20this%20email%20directly,%20view%20it%20on%20GitHub%3chttps:/urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497841731 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E__;IyU!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFoE5JACQ$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFlTwpxb4$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.>>

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

joeymoore commented 1 year ago

Sorry, there is no default proto anymore and you will need to either put TCP or UDP. IMPERVA_SYSLOG_PROTO= I can update this on the next release to default to UDP. Cheers, Joe

From: mahegs @.> Sent: Wednesday, April 5, 2023 11:40 AM To: imperva/incapsula-logs-downloader @.> Cc: Joe Moore @.>; State change @.> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Our previous Settings.Config:

APIID=XXXXXX APIKEY=XXXXXX SAVE_LOCALLY=NO PROCESS_DIR=/tmp/processed/ BASEURL=https://logs1.incapsula.com/3094_1202536/ USEPROXY=YES PROXYSERVER=http://proxy.fb:8080 SYSLOG_ENABLE=YES SYSLOG_ADDRESS=10.0.254.73 SYSLOG_PORT=514 USE_CUSTOM_CA_FILE=NO CUSTOM_CA_FILE= SYSLOG_PROTO=

And this is the updated version. Of note, we had a SAVE_LOCALLY=NO stanza in our previous config that I did carry over, but I ran it with and without it without a noticeable difference.

[SETTINGS] IMPERVA_API_ID=XXXXX IMPERVA_API_KEY=XXXXXX IMPERVA_API_URL=https://logs1.incapsula.com/3094_1202536/ IMPERVA_INCOMING_DIR= IMPERVA_PROCESS_DIR=/tmp/processed/ IMPERVA_ARCHIVE_DIR= IMPERVA_USE_PROXY=YES IMPERVA_PROXY_SERVER=http://proxy.fb:8080 IMPERVA_USE_CUSTOM_CA_FILE=NO IMPERVA_CUSTOM_CA_FILE= IMPERVA_SYSLOG_ENABLE=YES IMPERVA_SYSLOG_ADDRESS=10.0.254.73 IMPERVA_SYSLOG_PORT=514 IMPERVA_SYSLOG_PROTO= IMPERVA_SPLUNK_HEC= IMPERVA_SPLUNK_HEC_IP= IMPERVA_SPLUNK_HEC_PORT= IMPERVA_SPLUNK_HEC_TOKEN= IMPERVA_SPLUNK_HEC_SRC_HOSTNAME= IMPERVA_SPLUNK_HEC_INDEX= IMPERVA_SPLUNK_HEC_SOURCE= IMPERVA_SPLUNK_HEC_SOURCETYPE=

Thanks! Michal

From: Joe Moore @.> Sent: Wednesday, April 5, 2023 11:15 AM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Can you sure your config area for the SYSLOG info? Cheers, Joe

From: mahegs @.<mailto:@.>> Sent: Wednesday, April 5, 2023 10:08 AM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Joe Moore @.<mailto:@.>>; State change @.<mailto:@.>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

From: Joe Moore @.<mailto:@.>> Sent: Tuesday, April 4, 2023 3:12 PM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Hegstrom, Michal @.<mailto:@.>>; Mention @.<mailto:@.>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@mahegs<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E. You are receiving this because you were mentioned.Message ID: @.**@.https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E%20did%20you%20make%20the%20needed%20changes%20to%20the%20config%20file%20as%20all%20the%20variable%20names%20were%20changed%20to%20prepend%20the%20existing%20vars%20with%20IMPERVA_*%0b%0b-%0bReply%20to%20this%20email%20directly,%20view%20it%20on%20GitHub%3chttps:/urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497841731 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E__;IyU!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFoE5JACQ$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFlTwpxb4$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.>>

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497954110, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA. You are receiving this because you modified the open/close state.Message ID: @.***>

mahegs commented 1 year ago

Thank you for the assistance. I was able to get it working. However, I had to remove the IMPERVA_PROCESS_DIR=/tmp/processed/ value and use IMPERVA_PROCESS_DIR= to use the default for the logs to send to our syslog server. When I put a dir in for process logs the logs stop sending. I do not have enough space in the default process dir to store the logs to process and it fills up very quickly. Do you have anything I can try to get the logs to either compress sooner, or to better understand why the dir value I had before no longer works? Of note, the /tmp/processed/ dir is only 4 GB and I did not have issues with the previous script. The location it is writing to now has 9 GB, but it fills up extremely quickly with the 3.0 script.

Thanks! Michal

From: Joe Moore @.> Sent: Wednesday, April 5, 2023 12:46 PM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Sorry, there is no default proto anymore and you will need to either put TCP or UDP. IMPERVA_SYSLOG_PROTO= I can update this on the next release to default to UDP. Cheers, Joe

From: mahegs @.<mailto:@.>> Sent: Wednesday, April 5, 2023 11:40 AM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Joe Moore @.<mailto:@.>>; State change @.<mailto:@.>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Our previous Settings.Config:

APIID=XXXXXX APIKEY=XXXXXX SAVE_LOCALLY=NO PROCESS_DIR=/tmp/processed/ BASEURL=https://logs1.incapsula.com/3094_1202536/https://urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2daRvsr_B$ USEPROXY=YES PROXYSERVER=http://proxy.fb:8080 https://urldefense.com/v3/__http:/proxy.fb:8080__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dfHR7Jda$ SYSLOG_ENABLE=YES SYSLOG_ADDRESS=10.0.254.73 SYSLOG_PORT=514 USE_CUSTOM_CA_FILE=NO CUSTOM_CA_FILE= SYSLOG_PROTO=

And this is the updated version. Of note, we had a SAVE_LOCALLY=NO stanza in our previous config that I did carry over, but I ran it with and without it without a noticeable difference.

[SETTINGS] IMPERVA_API_ID=XXXXX IMPERVA_API_KEY=XXXXXX IMPERVA_API_URL=https://logs1.incapsula.com/3094_1202536/https://urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2daRvsr_B$ IMPERVA_INCOMING_DIR= IMPERVA_PROCESS_DIR=/tmp/processed/ IMPERVA_ARCHIVE_DIR= IMPERVA_USE_PROXY=YES IMPERVA_PROXY_SERVER=http://proxy.fb:8080 https://urldefense.com/v3/__http:/proxy.fb:8080__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dfHR7Jda$ IMPERVA_USE_CUSTOM_CA_FILE=NO IMPERVA_CUSTOM_CA_FILE= IMPERVA_SYSLOG_ENABLE=YES IMPERVA_SYSLOG_ADDRESS=10.0.254.73 IMPERVA_SYSLOG_PORT=514 IMPERVA_SYSLOG_PROTO= IMPERVA_SPLUNK_HEC= IMPERVA_SPLUNK_HEC_IP= IMPERVA_SPLUNK_HEC_PORT= IMPERVA_SPLUNK_HEC_TOKEN= IMPERVA_SPLUNK_HEC_SRC_HOSTNAME= IMPERVA_SPLUNK_HEC_INDEX= IMPERVA_SPLUNK_HEC_SOURCE= IMPERVA_SPLUNK_HEC_SOURCETYPE=

Thanks! Michal

From: Joe Moore @.<mailto:@.>> Sent: Wednesday, April 5, 2023 11:15 AM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Hegstrom, Michal @.<mailto:@.>>; Mention @.<mailto:@.>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Can you sure your config area for the SYSLOG info? Cheers, Joe

From: mahegs @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Sent: Wednesday, April 5, 2023 10:08 AM To: imperva/incapsula-logs-downloader @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Cc: Joe Moore @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>; State change @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

From: Joe Moore @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Sent: Tuesday, April 4, 2023 3:12 PM To: imperva/incapsula-logs-downloader @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Cc: Hegstrom, Michal @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>; Mention @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@mahegs<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E. You are receiving this because you were mentioned.Message ID: @.**@.<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E%20did%20you%20make%20the%20needed%20changes%20to%20the%20config%20file%20as%20all%20the%20variable%20names%20were%20changed%20to%20prepend%20the%20existing%20vars%20with%20IMPERVA_*%0b%0b-%0bReply%20to%20this%20email%20directly,%20view%20it%20on%20GitHub%3chttps:/urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***%3E%3E%3E

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497841731 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E__;IyU!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFoE5JACQ$https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E*3Chttps:/*urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E__;IyU!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFoE5JACQ$*3E__;IyUlLyoqJQ!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dUvy7Sjf$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFlTwpxb4$https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E*3Chttps:/*urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFlTwpxb4$*3E__;JSUvKiU!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dcgkvlkE$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

— Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$%3E. You are receiving this because you were mentioned.Message ID: @.**@.https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497954110 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497954110*3E__;IyU!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dQXbTfaw$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dWligHZV$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497960156__;Iw!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dYJdRwAK$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJD4RH7KG6FGLDU6PFLW7W4VRANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dSWAGToo$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

joeymoore commented 1 year ago

Ok, so this will be an interesting one to work though, so here is how the new process of downloading the logs and sending them works. The main thread will spawn a downloading thread, and this will have a few workers assigned to perform the downloading of the log. This is temporarily downloaded to the IMPERVA_INCOMING_DIR and then moved to IMPERVA_PROCESS_DIR. If either syslog or splunk is enabled, this will spawn another thread for watching the IMPERVA_PROCESS_DIR for incoming files. Once a new file is seen, a worker is set to send the log to either (syslog or splunk). Once successfully sent, there is a check for the IMPERVA_ARCHIVE_DIR to not be empty, this will cause zip and store the log in said dir.

All IMPERVA_*_DIR locations will default to the current working directory (where the script is running) and creating the associated directory where needed, i.e. IMPERVA_INCOMING_DIR will be {CWD}/incoming.

So let make sure all your locations are set correctly and then see if the archive/compression works fast enough. I can believe that the downloading process is faster than the however it's more likely that the archive is not happening fast enough. Please email me directly and we can jump on a zoom call. @.***

From: mahegs @.> Sent: Wednesday, April 5, 2023 2:46 PM To: imperva/incapsula-logs-downloader @.> Cc: Joe Moore @.>; State change @.> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Thank you for the assistance. I was able to get it working. However, I had to remove the IMPERVA_PROCESS_DIR=/tmp/processed/ value and use IMPERVA_PROCESS_DIR= to use the default for the logs to send to our syslog server. When I put a dir in for process logs the logs stop sending. I do not have enough space in the default process dir to store the logs to process and it fills up very quickly. Do you have anything I can try to get the logs to either compress sooner, or to better understand why the dir value I had before no longer works? Of note, the /tmp/processed/ dir is only 4 GB and I did not have issues with the previous script. The location it is writing to now has 9 GB, but it fills up extremely quickly with the 3.0 script.

Thanks! Michal

From: Joe Moore @.> Sent: Wednesday, April 5, 2023 12:46 PM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Sorry, there is no default proto anymore and you will need to either put TCP or UDP. IMPERVA_SYSLOG_PROTO= I can update this on the next release to default to UDP. Cheers, Joe

From: mahegs @.<mailto:@.>> Sent: Wednesday, April 5, 2023 11:40 AM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Joe Moore @.<mailto:@.>>; State change @.<mailto:@.>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Our previous Settings.Config:

APIID=XXXXXX APIKEY=XXXXXX SAVE_LOCALLY=NO PROCESS_DIR=/tmp/processed/ BASEURL=https://logs1.incapsula.com/3094_1202536/https://urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2daRvsr_B$ USEPROXY=YES PROXYSERVER=http://proxy.fb:8080 https://urldefense.com/v3/__http:/proxy.fb:8080__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dfHR7Jda$ SYSLOG_ENABLE=YES SYSLOG_ADDRESS=10.0.254.73 SYSLOG_PORT=514 USE_CUSTOM_CA_FILE=NO CUSTOM_CA_FILE= SYSLOG_PROTO=

And this is the updated version. Of note, we had a SAVE_LOCALLY=NO stanza in our previous config that I did carry over, but I ran it with and without it without a noticeable difference.

[SETTINGS] IMPERVA_API_ID=XXXXX IMPERVA_API_KEY=XXXXXX IMPERVA_API_URL=https://logs1.incapsula.com/3094_1202536/https://urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2daRvsr_B$ IMPERVA_INCOMING_DIR= IMPERVA_PROCESS_DIR=/tmp/processed/ IMPERVA_ARCHIVE_DIR= IMPERVA_USE_PROXY=YES IMPERVA_PROXY_SERVER=http://proxy.fb:8080 https://urldefense.com/v3/__http:/proxy.fb:8080__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dfHR7Jda$ IMPERVA_USE_CUSTOM_CA_FILE=NO IMPERVA_CUSTOM_CA_FILE= IMPERVA_SYSLOG_ENABLE=YES IMPERVA_SYSLOG_ADDRESS=10.0.254.73 IMPERVA_SYSLOG_PORT=514 IMPERVA_SYSLOG_PROTO= IMPERVA_SPLUNK_HEC= IMPERVA_SPLUNK_HEC_IP= IMPERVA_SPLUNK_HEC_PORT= IMPERVA_SPLUNK_HEC_TOKEN= IMPERVA_SPLUNK_HEC_SRC_HOSTNAME= IMPERVA_SPLUNK_HEC_INDEX= IMPERVA_SPLUNK_HEC_SOURCE= IMPERVA_SPLUNK_HEC_SOURCETYPE=

Thanks! Michal

From: Joe Moore @.<mailto:@.>> Sent: Wednesday, April 5, 2023 11:15 AM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Hegstrom, Michal @.<mailto:@.>>; Mention @.<mailto:@.>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Can you sure your config area for the SYSLOG info? Cheers, Joe

From: mahegs @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Sent: Wednesday, April 5, 2023 10:08 AM To: imperva/incapsula-logs-downloader @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Cc: Joe Moore @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>; State change @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

From: Joe Moore @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Sent: Tuesday, April 4, 2023 3:12 PM To: imperva/incapsula-logs-downloader @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Cc: Hegstrom, Michal @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>; Mention @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

@mahegs<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E. You are receiving this because you were mentioned.Message ID: @.**@.<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E%20did%20you%20make%20the%20needed%20changes%20to%20the%20config%20file%20as%20all%20the%20variable%20names%20were%20changed%20to%20prepend%20the%20existing%20vars%20with%20IMPERVA_*%0b%0b-%0bReply%20to%20this%20email%20directly,%20view%20it%20on%20GitHub%3chttps:/urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***%3E%3E%3E

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497841731 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E__;IyU!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFoE5JACQ$https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E*3Chttps:/*urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497841731*3E__;IyU!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFoE5JACQ$*3E__;IyUlLyoqJQ!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dUvy7Sjf$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFlTwpxb4$https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E*3Chttps:/*urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTGZWQZHP4TCFFNMXOZDW7WRJNANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFlTwpxb4$*3E__;JSUvKiU!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dcgkvlkE$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

— Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$%3E. You are receiving this because you were mentioned.Message ID: @.**@.https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497954110 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497954110*3E__;IyU!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dQXbTfaw$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dWligHZV$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497960156__;Iw!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dYJdRwAK$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJD4RH7KG6FGLDU6PFLW7W4VRANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dSWAGToo$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1498199751, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG4MZJEDVZF5U3V6NF3W7XR25ANCNFSM53AWOUDA. You are receiving this because you modified the open/close state.Message ID: @.***>

mahegs commented 1 year ago

From what I was seeing, it does seem the archive is not keeping up. At least not with the processing and storage we have available. Of note, when I started the 3.0 script and service, it started retrieving logs from a week ago. Is there a way to start with what is current, rather than so far back? I am wondering if the backlog it was trying to work through is part of the issue we are seeing.

A zoom call would be great! Your email address was masked in your message and only displayed as “@.<mailto:@.>”. I will try to format my email address a little creatively to see if you are able to see it to respond to me directly and we can go from there. michal.hegstrom at efirstbank dot com.

Thanks again! Michal

From: Joe Moore @.> Sent: Wednesday, April 5, 2023 4:22 PM To: imperva/incapsula-logs-downloader @.> Cc: Hegstrom, Michal @.>; Mention @.> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Ok, so this will be an interesting one to work though, so here is how the new process of downloading the logs and sending them works. The main thread will spawn a downloading thread, and this will have a few workers assigned to perform the downloading of the log. This is temporarily downloaded to the IMPERVA_INCOMING_DIR and then moved to IMPERVA_PROCESS_DIR. If either syslog or splunk is enabled, this will spawn another thread for watching the IMPERVA_PROCESS_DIR for incoming files. Once a new file is seen, a worker is set to send the log to either (syslog or splunk). Once successfully sent, there is a check for the IMPERVA_ARCHIVE_DIR to not be empty, this will cause zip and store the log in said dir.

All IMPERVA_*_DIR locations will default to the current working directory (where the script is running) and creating the associated directory where needed, i.e. IMPERVA_INCOMING_DIR will be {CWD}/incoming.

So let make sure all your locations are set correctly and then see if the archive/compression works fast enough. I can believe that the downloading process is faster than the however it's more likely that the archive is not happening fast enough. Please email me directly and we can jump on a zoom call. @.<mailto:@.>

From: mahegs @.<mailto:@.>> Sent: Wednesday, April 5, 2023 2:46 PM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Joe Moore @.<mailto:@.>>; State change @.<mailto:@.>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Thank you for the assistance. I was able to get it working. However, I had to remove the IMPERVA_PROCESS_DIR=/tmp/processed/ value and use IMPERVA_PROCESS_DIR= to use the default for the logs to send to our syslog server. When I put a dir in for process logs the logs stop sending. I do not have enough space in the default process dir to store the logs to process and it fills up very quickly. Do you have anything I can try to get the logs to either compress sooner, or to better understand why the dir value I had before no longer works? Of note, the /tmp/processed/ dir is only 4 GB and I did not have issues with the previous script. The location it is writing to now has 9 GB, but it fills up extremely quickly with the 3.0 script.

Thanks! Michal

From: Joe Moore @.<mailto:@.>> Sent: Wednesday, April 5, 2023 12:46 PM To: imperva/incapsula-logs-downloader @.<mailto:@.>> Cc: Hegstrom, Michal @.<mailto:@.>>; Mention @.<mailto:@.>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Sorry, there is no default proto anymore and you will need to either put TCP or UDP. IMPERVA_SYSLOG_PROTO= I can update this on the next release to default to UDP. Cheers, Joe

From: mahegs @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Sent: Wednesday, April 5, 2023 11:40 AM To: imperva/incapsula-logs-downloader @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Cc: Joe Moore @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>; State change @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Subject: Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Our previous Settings.Config:

APIID=XXXXXX APIKEY=XXXXXX SAVE_LOCALLY=NO PROCESS_DIR=/tmp/processed/ BASEURL=https://logs1.incapsula.com/3094_1202536/https://urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2daRvsr_B$https://urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/*3Chttps:/*urldefense.com/v3/__https:/logs1.incapsula.com/3094_1202536/__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2daRvsr_B$*3E__;JS8l!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJOihqrXL$ USEPROXY=YES PROXYSERVER=http://proxy.fb:8080 https://urldefense.com/v3/__http:/proxy.fb:8080__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dfHR7Jda$https://urldefense.com/v3/__http:/proxy.fb:8080*3Chttps:/*urldefense.com/v3/__http:/proxy.fb:8080__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dfHR7Jda$*3E__;JS8l!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJGOAzydk$ SYSLOG_ENABLE=YES SYSLOG_ADDRESS=10.0.254.73 SYSLOG_PORT=514 USE_CUSTOM_CA_FILE=NO CUSTOM_CA_FILE= SYSLOG_PROTO=

And this is the updated version. Of note, we had a SAVE_LOCALLY=NO stanza in our previous config that I did carry over, but I ran it with and without it without a noticeable difference.

Thanks! Michal

From: Joe Moore @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Sent: Wednesday, April 5, 2023 11:15 AM To: imperva/incapsula-logs-downloader @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Cc: Hegstrom, Michal @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>; Mention @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>> Subject: [EXTERNAL] Re: [imperva/incapsula-logs-downloader] Could not find file (Issue #46)

Can you sure your config area for the SYSLOG info? Cheers, Joe

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

With the Settings.Config updated, the logs are writing to /etc/incapsula/logs/config/logs.index, which is new, but they are not sending to our syslog server.

@mahegs<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E did you make the needed changes to the config file as all the variable names were changed to prepend the existing vars with IMPERVA_*

- Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E. You are receiving this because you were mentioned.Message ID: @.**@.<https://urldefense.com/v3/__https:/github.com/mahegs__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16cI_Rpb6$%3E%20did%20you%20make%20the%20needed%20changes%20to%20the%20config%20file%20as%20all%20the%20variable%20names%20were%20changed%20to%20prepend%20the%20existing%20vars%20with%20IMPERVA_*%0b%0b-%0bReply%20to%20this%20email%20directly,%20view%20it%20on%20GitHub%3chttps:/urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1496612538__;Iw!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fr4eqB0$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJF7622AJ4H7COYPSV3W7SFDXANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!Ah6suazwmKfBTWcbZtZ-Z2JAzVIc842MqV8hh_-ZJ3jB4hWIjmN8h4ByjZ1C3k3ChmH0XA-OAXJfsJ6jREiWO7E16fbjGTfX$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***%3E%3E%3E

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

— Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$%3E. You are receiving this because you were mentioned.Message ID: @.**@.<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497848587__;Iw!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFkrNRzG2$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJHWYMAYC2DTMOBM63LW7WSBDANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!B4Pujg6_l2w3WCR4EIwcDZ4EZ3-IDqv7cpYtV69D1-EFgKzw-bJ7VDA3F86px_bldpEcwUYuoXPHq3pgRt-jHSGYFh1pNRVk$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***%3E%3E%3E

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1497954110 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497954110*3E__;IyU!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dQXbTfaw$https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497954110*3E*3Chttps:/*urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497954110*3E__;IyU!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dQXbTfaw$*3E__;IyUlLyoqJQ!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJKkPi_Oi$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dWligHZV$https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA*3E*3Chttps:/*urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTG4C3454W2GTTUUGV4LW7W4BXANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dWligHZV$*3E__;JSUvKiU!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJAhrol2o$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>

— Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497960156__;Iw!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dYJdRwAK$%3E, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJD4RH7KG6FGLDU6PFLW7W4VRANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dSWAGToo$%3E. You are receiving this because you were mentioned.Message ID: @.**@.https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1497960156__;Iw!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dYJdRwAK$%3E,%20or%20unsubscribe%3chttps:/urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJD4RH7KG6FGLDU6PFLW7W4VRANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!HI08ob5gH6f2wgjCYtNAkmI_k1_STeLNXMNWAGUWcLFW5tM3RhniWFUa148kLzOIJrVvtJmHHjoFFpHynM-nCiz2dSWAGToo$%3E.%0bYou%20are%20receiving%20this%20because%20you%20were%20mentioned.Message%20ID:%20***@***.******@***.***>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/46#issuecomment-1498199751 https://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1498199751*3E__;IyU!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJJUCnfRb$, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG4MZJEDVZF5U3V6NF3W7XR25ANCNFSM53AWOUDA https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJZYTG4MZJEDVZF5U3V6NF3W7XR25ANCNFSM53AWOUDA*3E__;JQ!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJB3kmI4Z$. You are receiving this because you modified the open/close state.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/imperva/incapsula-logs-downloader/issues/46*issuecomment-1498233270__;Iw!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJPV8sFs6$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A3XGTJDDYJLXRPBZ72D2Q5LW7XV7DANCNFSM53AWOUDA__;!!EP3Ar9QBfU06DzGV!GfZ2LXS_V39hwYPNke7D37OPPeRmaTPF3hqT0WW6Jxe1nPO3ME9M6lbEHWEV5ps4EDRasEdQQ-a2266e7IqyFtsVJP8uD-5P$. You are receiving this because you were mentioned.Message ID: @.**@.>>

The information contained in this electronic communication and any document attached hereto or transmitted herewith is confidential and intended for the exclusive use of the individual or entity named above. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any examination, use, dissemination, distribution or copying of this communication or any part thereof is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this communication. Thank you.

imperva / incapsula-logs-downloader

Could not find file #46

python3 LogsDownloader.py