Marvin-Deng / Gemini-Scraper

A multithreaded web crawler powered by Google Gemini-Pro
2 stars 0 forks source link

Crawler returning nothing in some runs #17

Open Marvin-Deng opened 1 month ago

Marvin-Deng commented 1 month ago
image

When testing the endpoint running python3 tests/test_timing.py, sometimes nothing is returned. Try running the analyze topics and link extractor scripts in isolation, and check the outputs. I think the issue is something to do with the BFS part terminating early for some reason. Try adding additional error handling to see if you can fix the issue

Michaelyya commented 1 month ago

I'll run on this!