Rita does not detect new logs after the first import

[David-Aires]

Architecture

My project is made up of several docker containers, the DB (clickhouse), rita and zeek. A volume is shared between the Rita and Zeek containers in order to access the logs.

A cron is run every hour to import the logs into Rita. For the import, I only take the zeek logs folder of the day in order to have only 24h in Rita (the rolling option is also well set).

Issue

When Rita makes her first import of the day, everything goes according to plan. All the logs are found and imported.

2024-07-31T01:20:01Z INF Initiating new import... dataset=my_dataset directory=/app/zeek-logs/2024-07-31 rebuild=false rolling=true started_at="2024-07-31 01:20:01.350964382 +0000 UTC m=+0.004145583"
2024-07-31T01:20:01Z INF [THREAT INTEL] Updating online feed... feed_url=https://feodotracker.abuse.ch/downloads/ipblocklist.txt
[-] Parsing:  /app/zeek-logs/2024-07-31/conn.00:10:00-00:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/conn.00:30:00-00:40:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/conn.00:20:00-00:30:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/conn.00:40:00-00:50:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/conn.00:50:00-01:00:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.00:10:00-00:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.00:20:00-00:30:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.00:30:00-00:40:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.00:40:00-00:50:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.00:50:00-01:00:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.00:10:00-00:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.00:20:00-00:30:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.00:30:00-00:40:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.00:40:00-00:50:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.00:50:00-01:00:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.00:10:00-00:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.00:20:00-00:30:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.00:30:00-00:40:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.00:40:00-00:50:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.00:50:00-01:00:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.00:10:00-00:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.00:20:00-00:30:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.00:30:00-00:40:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.00:40:00-00:50:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.00:50:00-01:00:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.00:10:00-00:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.00:20:00-00:30:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.00:30:00-00:40:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.00:40:00-00:50:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.00:50:00-01:00:00.log.gz
2024-07-31T01:20:08Z INF Finished Parsing Logs! 🎉 elapsed_time=7.308483578s parsing_began=1722388801 parsing_finished=1722388808

🧂 Seasoning open SSL connections  🎉   100%

🧂 Seasoning SSL connections       🎉   100%

🧂 Seasoning HTTP connections 🎉   100%

✅ Sifting open IP connections...

✅ Sifting IP connections...

2024-07-31T01:20:09Z INF Finished Seasoning Logs! 🎉 elapsed_time=595.68529ms seasoning_began=1722388808 seasoning_finished=1722388809

SNI Connection Analysis 🎉   100%

IP Connection Analysis  🎉   100%

DNS Analysis            🎉   100%

2024-07-31T01:20:11Z INF Finished Analysis! 🎉 analysis_began=1722388809 analysis_finished=1722388811 elapsed_time=1.736611685s
2024-07-31T01:20:11Z INF Finished Modification! 🎉 elapsed_time=32.84501ms modification_began=1722388811 modification_finished=1722388811
2024-07-31T01:20:11Z INF Finished Importing Hour Chunk day=0 elapsed_time=9.732068841s hour=0
[-] Parsing:  /app/zeek-logs/2024-07-31/conn.01:00:00-01:10:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.01:00:00-01:10:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/http.01:10:00-01:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.01:00:00-01:10:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/ssl.01:10:00-01:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.01:00:00-01:10:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_conn.01:10:00-01:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.01:00:00-01:10:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/open_ssl.01:10:00-01:20:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.01:00:00-01:10:00.log.gz
[-] Parsing:  /app/zeek-logs/2024-07-31/dns.01:10:00-01:20:00.log.gz
2024-07-31T01:20:13Z INF Finished Parsing Logs! 🎉 elapsed_time=2.082223516s parsing_began=1722388811 parsing_finished=1722388813

🧂 Seasoning open SSL connections  🎉   100%

🧂 Seasoning SSL connections       🎉   100%

🧂 Seasoning HTTP connections 🎉   100%

✅ Sifting open IP connections...

✅ Sifting IP connections...

2024-07-31T01:20:13Z INF Finished Seasoning Logs! 🎉 elapsed_time=467.513435ms seasoning_began=1722388813 seasoning_finished=1722388813

SNI Connection Analysis 🎉   100%

IP Connection Analysis  🎉   100%

DNS Analysis            🎉   100%

2024-07-31T01:20:15Z INF Finished Analysis! 🎉 analysis_began=1722388814 analysis_finished=1722388815 elapsed_time=1.889514043s
2024-07-31T01:20:15Z INF Finished Modification! 🎉 elapsed_time=24.338697ms modification_began=1722388815 modification_finished=1722388815
2024-07-31T01:20:15Z INF Finished Importing Hour Chunk day=0 elapsed_time=4.591641917s hour=1
2024-07-31T01:20:15Z INF 🎊✨ Finished Import! ✨🎊 elapsed_time=14.6s

But on subsequent imports, Rita no longer imports anything and reports that all the files have already been imported.

2024-07-31T02:20:16Z INF Initiating new import... dataset=my_dataset directory=/app/zeek-logs/2024-07-31 rebuild=false rolling=true started_at="2024-07-31 02:20:16.236288652 +0000 UTC m=+0.004589947"
2024-07-31T02:20:16Z INF [THREAT INTEL] Updating online feed... feed_url=https://feodotracker.abuse.ch/downloads/ipblocklist.txt

[!] all files were previously imported

2024-07-31T03:20:16Z INF Initiating new import... dataset=my_dataset directory=/app/zeek-logs/2024-07-31 rebuild=false rolling=true started_at="2024-07-31 03:20:16.603033134 +0000 UTC m=+0.006243644"
2024-07-31T03:20:16Z INF [THREAT INTEL] Updating online feed... feed_url=https://feodotracker.abuse.ch/downloads/ipblocklist.txt

[!] all files were previously imported

2024-07-31T04:20:16Z INF Initiating new import... dataset=my_dataset directory=/app/zeek-logs/2024-07-31 rebuild=false rolling=true started_at="2024-07-31 04:20:16.98451503 +0000 UTC m=+0.004978521"
2024-07-31T04:20:17Z INF [THREAT INTEL] Updating online feed... feed_url=https://feodotracker.abuse.ch/downloads/ipblocklist.txt

[!] all files were previously imported

...

However, there are new logs after the first ones that have been imported...

[lisaSW]

You are importing the same folder that you had in your first import, are your new logs in a new folder for the next day?

[David-Aires]

Yes, each day has its own log folder. But I want to import the same day's logs every hour, so the folder must remain the same throughout the day. Rita needs to detect new zip logs in the folder.

In the following image, I have new logs (for the same day) arriving as time goes by. If I run Rita in the morning, the tool imports the morning's logs, but not the afternoon's if I run it at the end of the day. Logically, RITA should detect these new compressed logs and import them. Exactly as in the old version.

[lisaSW]

Issue was addressed and release updated, please get a fresh copy of v.5.0.7 and let us know if it works.

[David-Aires]

Everything's working properly now! Many thanks for the fix :heart_eyes: and for this incredible tool.

[Mohammad-Mirasadollahi]

Hi I have the same problem This problem still exists in v5.0.8