Open dylangrech92 opened 8 years ago
Due to browser limitations, the crawler can only check up to 2 urls at a time however PHP can perform several curls at once.
The batch size should be a config as different users might want to only curl 1 page at a time due to server load. Default: 1 curl at a time
Due to browser limitations, the crawler can only check up to 2 urls at a time however PHP can perform several curls at once.
The batch size should be a config as different users might want to only curl 1 page at a time due to server load. Default: 1 curl at a time