JM / AEO race condition on worker registration

[cortlandstarrett]

As part of #219 (Application Messaging Overrun), we run a scenario where audit events are already waiting in the message broker when the application launches. This leads to the following race condition.

• Worker requests registration.
• JM accepts and acknowledges successful registration.
• JM then assigns a job to this worker and sends an audit event.
• The audit event gets to the worker before, or at the same time, as the registration acknowledgement.
• The Worker gets the event before it has transitioned into Registered.

The worker then deregisters and re-registers and sometimes syncs back up.

[cortlandstarrett]

I note that JobManager.selectWorkerForJob does not interrogate the 'working' attribute. Adding this interrogation could add an additional condition to the assignment.

[cortlandstarrett]

In worker state 'Registered' the heartbeat timer could be set to 0 seconds rather than a full duration to cause the very first heartbeat to occur quickly. This way JM could use the heartbeat to move the EmployedWorker state machine along.

[cortlandstarrett]

{"timestamp":"2024-06-11T18:00:44.339Z","payload":{"eventId":"","eventName":"","jobId":"","jobName":"","message":"received event for unregistered worker","tag":"aeordering_rcvd_unregistered","workerId":"1ab1cf0b-7a4e-4466-a9eb-d07e8bac13f7"}}

[cortlandstarrett]

fixed with #225