search

commoncrawl / news-crawl

News crawling with StormCrawler - stores content as WARC
Apache License 2.0
316 stars 34 forks source link

Port topology and resources to StormCrawler 2.10 #60

Open jnioche opened 9 months ago

jnioche commented 9 months ago

Upgrade Apache Storm, ElasticSearch and Kibana

This way the NewsCrawler will benefit from the many bugfixes and improvements provided by these components and make it easier ti add new functionalities going forward.

alextechnology commented 9 months ago

Hello - I posted a comment in Discussions regarding 2.x not working with multiple topology workers

jnioche commented 9 months ago

Hello - I posted a comment in Discussions regarding 2.x not working with multiple topology workers

thanks @alextechnology I'll have a look early next week