search

GreyWyvern / orca-search2

Orca Search website search engine
GNU General Public License v3.0
1 stars 0 forks source link

Incorrect key to initiate spider #5

Open rothkj1022 opened 5 years ago

rothkj1022 commented 5 years ago

I'm running a full scan on a large site, hosted with PHP-FPM (7.2), and after some time, I get the message "Incorrect key to initiate spider" via email. I'm guessing that I'm reaching a timeout.

Also, unfortunately with PHP-FPM, I cannot see the crawl status screen, as output buffering cannot be altered to my understanding. Is there something that can be done about this as well?

GreyWyvern commented 5 years ago

How large is the site? That's unusual since the the only time (knock wood) that error could happen is right at the beginning of spider execution. On line 525 of spider.php, you might want to add more verbose output to $_XDATA['errors'] to see why execution is reaching it.

As for the progressive output buffering, the real fix would be to have the output screen fetching the progress from the server via JS instead of buffering HTML at all. It's a bigger change, but when I first built the search script, I wasn't all that familiar with Ajax. :)

rothkj1022 commented 5 years ago

This particular site (www.afpgusa.com) has a couple hundred urls to crawl. It does seem that the spidering does finish, despite getting the error at the start. I'm going to ignore this for now and dig deeper one day when I can find time.