Rethink worker tasks & dependencies

[anuejn]

Maybe it makes sense to make the dependencies more granular. E.g. the alignment could start aligning to the already transcribed bars (maybe with some safety distance) even if not the whole document is transcribed yet. This could also lead to some cool UX, where we really display one worker for each task (similiar to normal users) and each worker can report their own status

[pajowu]

IMO the better way for now would be to split the tasks into smaller tasks. For example we already agreed that we should split the current ALIGN into two: one task that only does alignment and only depends on the transcript. And a second task that applies the diarization results to the re-aligned segments.

I fear that running multiple tasks in parallel that depend on each other could introduce a lot of complexity for relatively small gains. Some things we would maybe need (just brainstorming)

• A way for the dependent task to know when it can start and which part of the document is already safe to work with
• A method to apply incoming changes to the automerge document in the worker
• A method to deal with changes in the document we didn't expect (e.g. what if a part we already aligned changes, what if during alignment the part we are aligning changes?)
• A way to signal failing jobs (so the waiting job can stop running)
• A way to run multiple jobs a the same time on a worker / how a job can yield for a certain amount of time (bonus question: how do we handle a job crashing in a worker with >1 job?)

IMO a solution might be for tasks to schedule their dependants (e.g. the transcription task spawning new alignment tasks), which is not the fastest solution and also poses some new UI/UX problems. But might work safer with the current architecture