Option 1:
Enable a way in the python control script to indicate the need to kill the harvest, but do the post processing. Can be done by monitoring for a STOP file, if seen kill the harvest and proceed with the post processing.
Option 2:
The harvester catches SIGUSR1, stops, and just exists with 0, so post processing continuous.
Option 3:
An overall timeout - if a harvesting thread takes more than it just gets terminated (e.g. monitoring thread interrupts it). Maybe JMX or another mechanism can be used to tweak these timeouts so that a running harvest can be meddled with.
java.net.HttpURLConnectionmay hang is special cases (see https://stackoverflow.com/questions/15874834/httpclient-hangs-on-socketread0-with-successfully-executed-method/19189036#19189036). No timeouts are triggered, so the only thing left is killing the harvest, but this skips post processing.Option 1: Enable a way in the python control script to indicate the need to kill the harvest, but do the post processing. Can be done by monitoring for a
STOPfile, if seen kill the harvest and proceed with the post processing.Option 2: The harvester catches
SIGUSR1, stops, and just exists with0, so post processing continuous.Option 3: An overall timeout - if a harvesting thread takes more than it just gets terminated (e.g. monitoring thread interrupts it). Maybe JMX or another mechanism can be used to tweak these timeouts so that a running harvest can be meddled with.