implement Scheduler._timeout handler

[jack-oquin]

Currently stubbed out.

[jack-oquin]

My current timeout implementation simply deletes the corresponding _RequesterStatus entry from the scheduler's dictionary, which makes sense, since contact has been lost.

Advantages:

• Simple
• No additional state transitions
• Idempotent protocol: every SchedulerRequests message has essentially the same meaning
• Resources can be cleaned up reliably

Issues:

• Need another callback for the scheduler to free any resources it had allocated to that requester
• No memory of any previous activity if the same requester tries to reconnect
• New connections have some special semantics: any state other than NEW or RESERVED should trigger a PREEMPTED reply with reason=TIMEOUT.

[jack-oquin]

Looks like no new scheduler callback is required:

• set the state of every non-CLOSED request to CANCELING with reason TIMEOUT
• invoke the normal scheduler callback to do its usual processing

This essentially treats the timeout as if a message had been received canceling every active request.

[jack-oquin]

Another advantage to this approach:

• Restarting a requester works the same whether it adopts a new UUID or retains the old one.

[jack-oquin]

I am quite happy with the way this turned out.

Closing now, re-open if design problems crop up.