Fix Worker ref until worker is actually removed

[tj]

this.removeWorker(worker.id); is called when removing a worker, however just because it is signaled does not mean it's dead, so subsequent events related to the worker will not map properly