HighThroughputExecutor workers can connect to somebody else's interchange

[benclifford]

Describe the bug There is no validation that a high throughput executor worker is connected to the right place: as long as whatever it connects to speaks the protocol well enough, it will connect. Likewise, there is no validation that an interchange is sending tasks to the right worker: as long as someone has connected and speaks the right protocol well enough, they will be sent work.

This is both an impediment to successful runs, in a few different places: I've personally encountered it in the funcx fork of htex and in parsl's local CI testing.

It is also a security vulnerability: anything sensitive in invocation parameters can be disclosed to whoever connects; and whoever connects can return arbitrary false results.

Expected behavior interchange and workers should have a higher degree of trust that a connection is true.

[daheise]

In order for this issue to cause a problem, do the HighThroughputExecutors have to match each other in configuration, or will they all cross talk arbitrarily? Is there any work around to prevent accidental (as opposed to malicious) cross talk?

[benclifford]

They would need to match in configuration in the sense of being configured with the same network ports. Nothing more than that, though.

I hope to be able to implement something simple (eg with random tokens) to prevent accidental cross talk, in the next few months.

[benclifford]

crossref #952

[benclifford]

I've observed, and continue to observe, this happening in the CI test suite and causing ongoing test failures.