search

jungjonghun / crawler4j

Automatically exported from code.google.com/p/crawler4j
0 stars 0 forks source link

Count of crawl cycles #305

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Hi,
Is there a way to count the number of complete crawl cycle?

Thanks!!

Original issue reported on code.google.com by edgar.ri...@gmail.com on 15 Sep 2014 at 4:21

GoogleCodeExporter commented 9 years ago 
Please explain what is a crawl cycle ?

Do u mean crawl depth ?

Original comment by avrah...@gmail.com on 15 Sep 2014 at 4:45

GoogleCodeExporter commented 9 years ago 
Hi,
The Crawl Cycle, its when the crawler START and END, basically.

If I run again the crawler it would increment the number of crawl cycle.

If you stop the crawler and restart it again it would hold the current crawl 
cycle number, because the cycle is not complete it yet.

Regards

Original comment by edgar.ri...@gmail.com on 15 Sep 2014 at 2:06

GoogleCodeExporter commented 9 years ago 
Sorry, I still don't understand

You set resumable to true ?
Then you crawl again and it continues using the remaining seeds.

I don't see where the cycles get inside ?

Original comment by avrah...@gmail.com on 15 Sep 2014 at 2:10

GoogleCodeExporter commented 9 years ago 
Hi,
When I run the crawler, at some point it will finish, just count the number of 
times that the crawler has been run

Original comment by edgar.ri...@gmail.com on 15 Sep 2014 at 3:03

GoogleCodeExporter commented 9 years ago 
Just create a basic file and write a "0" in it. 
In java: Open this file before you start crawling. Read the number from the 
file and increase it by one. Then save the file and it will count up on every 
crawl run.

Original comment by tobias.k...@web.de on 16 Sep 2014 at 2:23

GoogleCodeExporter commented 9 years ago 
This question is better served in the forum.

You can use the docId I suppose, but best for you to do as Tobias has suggested 
and count for yourself using your DB or file.

Just don't forget to configure resumable crawling to true.

I am closing this as this is not a bug...

Original comment by avrah...@gmail.com on 5 Dec 2014 at 8:54