Also introduces emit_heartbeat function on context object. The aim is to expose a way for the crawlers to indicate that they are alive. It should be particularly useful if the crawler has a huge for loop that take hours to run like many of our bigger crawlers.
Also introduces
emit_heartbeatfunction oncontextobject. The aim is to expose a way for the crawlers to indicate that they are alive. It should be particularly useful if the crawler has a huge for loop that take hours to run like many of our bigger crawlers.