Closed kaal-zathura24 closed 4 years ago
Well, that fetches fail and show up as FETCH_ERROR in the status index isn't unexpected, there are many possible reasons. Could you check the crawler logs (usually /var/log/storm/workers-artifacts/NewsCrawl*/*/worker.log
) for errors? Every fetch is logged and if it fails the reason should be given there.
Also important: the project does not index content into elasticsearch, only the status index is filled. The content is stored in WARC files placed in /data/warc/ by default. If you want to index the news content, that's possible but it won't be part of this project. Have a look at the documentation of stormcrawler to figure out how you need to modify the crawler topology to index page content.
thanks a lot , that helped
From: Sebastian Nagel notifications@github.com Sent: 11 June 2020 02:40 To: commoncrawl/news-crawl news-crawl@noreply.github.com Cc: 3342 GAURAV KUMAR gkumar_17291@aitpune.edu.in; Author author@noreply.github.com Subject: Re: [commoncrawl/news-crawl] unable to fetch data from elasticsearch , no content is showing (#38)
Well, that fetches fail and show up as FETCH_ERROR in the status index isn't unexpected, there are many possible reasons. Could you check the crawler logs (usually /var/log/storm/workers-artifacts/NewsCrawl//worker.log) for errors? Every fetch is logged and if it fails the reason should be given there. Also important: the project does not index content into elasticsearch, only the status index is filled. The content is stored in WARC files placed in /data/warc/ by default. If you want to index the news content, that's possible but it won't be part of this project. Have a look at the documentation of stormcrawler to figure out how you need to modify the crawler topology to index page content.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/commoncrawl/news-crawl/issues/38#issuecomment-642267388, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AP5IOSZF7OS3PSFOBGVMLI3RV7ZDLANCNFSM4N2WMESQ.
i have used the following github repository "https://github.com/commoncrawl/news-crawl" he has used the following versions of required libraries
Install Elasticsearch 7.5.0 Install Apache Storm 1.2.3 stormcrawler 1.16 maven 3.6.2
I have followed the path given in readme but my localhost:9200 is not showing any hits the configuration command is successfully running but it shows some FETCH_ERROR errors. No content or url is being shown.