LogsDownloader script needs restart in order to collect/forward logs

[fedon99]

Hi all,

We are using LogsDownloader script on a syslog server and we forward the logs to our SIEM infrastructure. After each 4-5 hours logs are not appearing on SIEM and we need to restart the service on the syslog server to start fetching and forward the events again.

Have you experienced an issue like this?

Thank you in advnace.

[joeymoore]

@fedon99 please use the beta 3.0 and this will resolve this issue. The 3.0.0 release will be the main release very soon. https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta

[fedon99]

So this is a common issue for version that are older than 3.0 beta ?

[joeymoore]

Yes, the logs are produces too fast and the downloader process, previously single threaded, could not keep up.

[lachlanjholmes]

@fedon99 did you update the code fix your issue; I'm running 3.0.0 beta code that is live in the master branch right now and it's still falling over after 4 hours or so.

[fedon99]

@lachlanjholmes we will test the new version in the coming days and I will let you know. So the same problem appears to the new version also @joeymoore ?

[fedon99]

Hi @joeymoore ,

Also the link that you mentioned above is not working anymore.

https://github.com/imperva/incapsula-logs-downloader/tree/release-3.0.0-beta

Have the new version became the main release?

Thank you.

[joeymoore]

We released 3.0 this last week, so a normal pull will be the new code.

[fedon99]

Hi @lachlanjholmes,

for 4 days we have no issues.

But @joeymoore. For example, I checked 2 IP's and I found that we received everyday the same logs, even though the events have happend since 2 days ago? Do you know anything about that?

[joeymoore]

Hello @fedon99 , happy to hear that things are working better and to answer your question about the duplicates; this will happen until the new downloader catches up to the log index number that was last downloaded by the old version. This was something that we could not prevent in the upgrade with the improved logging integrity. The new downloader will download ever log in the index file and this will very for upgrading users verse new users. Apologize for the annoyance but this will clear. Cheers

[fedon99]

Thank you for your prompt response. This should be expected to take days?

[joeymoore]

Look in the config dir and see the difference between logs.index and complete.log, once these match, you should not have any more duplicates. Remembering that the complete.log will continue to grow and the logs.index will dropped off older indexes over time; this represents what in the log store where we are downloading from.

[lachlanjholmes]

I've got it working now in on a plain VM with the 3.0 code.

I was trying to get it working in docker but the lack of output to standardout made me believe that the container was silently failing. To be fair I still dont know if it works or not in docker. This will be something I will have to look more at.

Maybe persistant volumes is needed for that file that tracks what the last successfully file. As I've noticed it just restarts from the oldest file on the logs url.

Maybe an update to the dockerfile is needed for this.

Hey @joeymoore is it possible to get a systemd unit file documented as well? I've done my own but it would be good to have it from someone at Imperva.

Also -v or -c or any of the switches don't seem to work... it just doesn't notice the path/file I give it...

EDIT: spelling

[fedon99]

@joeymoore I am seeing for the same IP two type of logs one that is containing the date and one with no date

<30> Nov 28 10:54:41 "hostname" cwaf CEF:0|Incapsula|SIEMintegration|1|1|IncapRules..... <14>CEF:0|Incapsula|SIEMintegration|1|1|IncapRules.... That is what you are saying? The new one is with the date and the old one without the date on the log? Thank you

[joeymoore]

This is correct, the newer has the date time.

[joeymoore]

I've got it working now in on a plain VM with the 3.0 code.

I was trying to get it working in docker but the lack of output to standardout made me believe that the container was silently failing. To be fair I still dont know if it works or not in docker. This will be something I will have to look more at.

Maybe persistant volumes is needed for that file that tracks what the last successfully file. As I've noticed it just restarts from the oldest file on the logs url.

Maybe an update to the dockerfile is needed for this.

Hey @joeymoore is it possible to get a systemd unit file documented as well? I've done my own but it would be good to have it from someone at Imperva.

Also -v or -c or any of the switches don't seem to work... it just doesn't notice the path/file I give it...

EDIT: spelling

I will investigate this.

[fedon99]

Hi @joeymoore

Some logs seems to not have src field. Why this is happening?

Thank you.

[joeymoore]

Hi @felon99, can we jump on a zoom call to help review some of your questions? Email me at @.*** and we'll set something up. Cheers, Joe

---

From: fedon99 @.> Sent: Thursday, November 30, 2023 6:31 AM To: imperva/incapsula-logs-downloader @.> Cc: Joe Moore @.>; Mention @.> Subject: Re: [imperva/incapsula-logs-downloader] LogsDownloader script needs restart in order to collect/forward logs (Issue #65)

CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi @joeymoorehttps://github.com/joeymoore

Some logs seems to not have src field. Why this is happening?

Thank you.

— Reply to this email directly, view it on GitHubhttps://github.com/imperva/incapsula-logs-downloader/issues/65#issuecomment-1833889904, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJZYTG3GEH2OL2WM4YU7YDLYHCKFRAVCNFSM6AAAAAA67GITG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTHA4DSOJQGQ. You are receiving this because you were mentioned.Message ID: @.***>

---

This message is confidential. If you believe you received this message in error, please inform the sender and delete this message and all attachments.

[joeymoore]

I added my email to my profile here: https://github.com/joeymoore

[fedon99]

Hi @joeymoore,

I resolved this issue by change the config file to transfer the logs through TCP not UDP. Now I think all the problems have been solved. We are going to deploy the new script everywhere and we will monitor the logs ingestion. If there is a problem again I will write a comment or I will send you a message.

Thank you for your support.

[G4fanhoto]

Good evening friends, did you solve this problem with sending LOGs? Are you using UDP or TCP, I have this problem when forwarding logs to the SIEM I am using this new script

@fedon99 @joeymoore

[fedon99]

@G4fanhoto We are using TCP. Yes with the use of the new script we have no issues. What problem do you have exactly?

[G4fanhoto]

@G4fanhotoEstamos usando TCP. Sim, com o uso do novo script não temos problemas. Que problema você tem exatamente? @fedon99 I have a problem with logs not being forwarded to SIEM with a new script, I changed it to TCP and it didn't solve it. I use a proxy to access the internet to consume the incapsula logs

[joeymoore]

@G4fanhoto is the proxy required to send to your SEIM? We don't have this capability to use a proxy for the "send" functions.

[G4fanhoto]

@G4fanhoto is the proxy required to send to your SEIM? We don't have this capability to use a proxy for the "send" functions.

@joeymoore Good afternoon Joey, thanks for getting back to us.

I use a proxy to make calls to the internet and consume logs in SaaS (https://logs.incapsula.xxxx) it is only used for this request.

We do not use a proxy to send syslog

[joeymoore]

Thanks @G4fanhoto - can I ask what type of logs your using in the configuration? I have seen this issue when users use the W3C or LEEF format?

[G4fanhoto]

Thanks @G4fanhoto - can I ask what type of logs your using in the configuration? I have seen this issue when users use the W3C or LEEF format?

We are using CEF

[joeymoore]

@G4fanhoto please go to my github profile (github.com/joeymoore) and email me directly. We can jump on a zoom and resolve this.

[joeymoore]

@G4fanhoto Please email me directly from my GitHub profile, I will close this issue. https://github.com/joeymoore