Open anjackson opened 4 years ago
Implemented a flush
and added support for a close
to crawl-streams
. Will update and trial.
Hmm, code for crawl-streams
includes a refreshDepth
in the launcher. Need to check if that's ready to go as far as Heritrix is concerned and then review the launcher/enqueue implementation.
After noticing a small crawl gap:
i.e. the KafkaProducer is being closed after the task has completed, but has not flushed.