Describe the bug
In some situations, the interchange will go away. Commonly these are: OOM killer, user pressing ctrl-C, developers hacking on the interchange and breaking it.
In this situation, the submit side sits waiting for the interchange to reappear on ZMQ channels, but neither detects it's gone nor attempts any repairing action.
This kind of failure means that high throughput executor can not continue to work, and the submit side should act accordingly, rather than hanging
To Reproduce
kill the interchange midway through a test run
Describe the bug In some situations, the interchange will go away. Commonly these are: OOM killer, user pressing ctrl-C, developers hacking on the interchange and breaking it.
In this situation, the submit side sits waiting for the interchange to reappear on ZMQ channels, but neither detects it's gone nor attempts any repairing action.
This kind of failure means that high throughput executor can not continue to work, and the submit side should act accordingly, rather than hanging
To Reproduce kill the interchange midway through a test run