Provide a forced shutdown option.

[zhouaihui]

What happened

Demo code.

@fed.remote
def error_fun():
    raise Exception('By design')

@fed.remote
def random():
    return [1, 2] 

@fed.remote
def foo(a):
    print(a)

try:
    # Alice ran into an error and broadcast error to bob. And exit then.
    a = error_fun.party('alice').remote()
    b = foo.party('bob').remote(a)

    # Alice did not execute the following codes.
    data = random.party('alice').remote()
    c = foo.party('bob').remote(data)
   # Bob was going to send c to alice but alice won't send `data` to bob since alice exited already. 
    foo.party('alice').remote(c)

   # Bob got the error.
    fed.get(b)

finally:
    # Bob was blocked here because data sending was hanging.
    fed.shutdown()

A possible solution

Give an forced option to fed.shutdown(forced=False), if forced==True, shutdown anyway ignoring the data sending.

[jovany-wang]

fed.shutdown(forced=False) or fed.force_shutdown() ?

[zhouaihui]

We can hide the forced behavior? Once bob received error from alice, the job became meaningless already. So bob can mark the job as failure when received an error from alice, and then do not wait for data sending when shutdown.

[zhouaihui]

Receiving an error from peer and Error on self data sending are same case actually. We should not wait data sending if error occurred no matter from alice itself or from bob, since a job became meaningless already.

[jovany-wang]

Receiving an error from peer and Error on self data sending are same case actually. We should not wait data sending if error occurred no matter from alice itself or from bob, since a job became meaningless already.

+1