Better halting solution

[krtek4]

35 is now closed, but I am still not satisfied with the solution. We wait on all currently processing URLs but no report is generated at the end.

I think we could improve this a lot by decoupling the crawler from the tester a bit more :

• Each has its own queues and knows only about the orchestrator
• The "orcherstrator" is the only one knowing about maximumUrls
• The crawler sends new URL to the orchestrator which can stop it if the maximum is reached
• The orchestrator then populates the tester queue
• The tester queue is not "killed" upon the first Ctrl-C but a flag is merely set which only tells to not process the remaining URLs in the queue

This way, we will probably fix a race condition that still arise sometimes where the report of one of the URL has not the time to be written because the last item is finished and node exits.

We could also have better exit code management.

And last but not least, URLs currently being processed will have time to generate a report. Only the second Ctrl-C will stop the process immediately.

(And I forget to mention a cleaner implementation)

[Hywan]

Damn, you did a huge refactoring without any PR. That's not fair.

[krtek4]

My cat rewied it ;)

[krtek4]

Or reviewed depends on how many vodka he had yesterday :p

[Hywan]

I guess the halt current solution is fine. Can we close this issue?

[krtek4]

Since I opened this issue after implementing the current solution I would say no.

[krtek4]

Oh sorry, didn't see you made changes regarding this in a totally unrelated PR.

OK then

[Hywan]

:+1: