Update the crawl condition when continuing the loop early

[catalystfd]

We've encountered pathological issues where the crons appear to get stuck within these loops and never ever finish, which leads to the rest of the site tasks being starved of resource.