Problem with web crawling [stops on pending status without crawling]

medialab / hyphe

Websites crawler with built-in exploration and control web interface

http://hyphe.medialab.sciences-po.fr/demo/

GNU Affero General Public License v3.0

329 stars 59 forks source link

Problem with web crawling [stops on pending status without crawling] #417

Closed maaxpamp closed 3 years ago

maaxpamp commented 3 years ago

Hi!

I've been trying to crawl another part of discovered websites. When I run crawling, the application does not crawling them. There's information on the upper "refreshing crawl job", but nothing is happening. I cancelled the crawling of the bundle of websites twice. I tried to recrawl the sites individually, but it does not work either.

In the past there I didn't have any problem with crawling. Any ideas on how to fix it?

Best regards, Marcin

boogheta commented 3 years ago

Hello, Can you indicate how/where you are using hyphe? Is this from the online demo? From a local installation? And does the problem still happen as of today?

Agnes-researcher commented 3 years ago

Hello

I also encountered this problem since yersterday. I use hyphe Browser from the demo server. Thanks

Fortunat-Fr commented 3 years ago

The same here... Doesn't Work! I am working from the online demo. I got stuck on Crawling page

boogheta commented 3 years ago

Hello all, There was indeed an issue on the demo instance that was freezing crawls. I restarted the containers and it caught up on all pending crawls, everything should be working fine now.