Optimize computing progress

[KrzysztofMadejski]

How do we generally show progress: 1) Should it give an idea how much work has been started? (but not crosschecked?) 2) Should it give an idea how much work has been finished? (it is crossverified) Let's start with (2) and maybe in the future show also progress of those "in progress", which will have higher values.

Progress overall is average progress across all docs.

Progress of a document is an average total progress of all of tasks created on such document.

Own progress of a task is:

• 100% if cross-checked positively
• 95% * (1 - e^{-2 \cdot \frac{1}{entries_to_run_crosscheck}(no_entries)})

Total progress (with children) of a task is:

• own progress if task won't have any children
• general ratio: ( average(subtasks progress) * average_number_of_subtasks(configured in task) + own_progress ) / (average_number_of_subtasks(configured in task) + 1)
• above has two distinct values, because when own_progress != 1 then subtasks_progress == 0

UI:

• [ ] progress of "dirty task" should be colored differently

? How to compute that effectively?

1. In some background job working from bottom up.
   1. create a tree of task types
   2. compute and cache progress for all tasks and all levels starting at the lowest
   3. compute progress for docs
   4. compute overall progress
2. update those when value changes for a single task? Maybe yes to get instant feedback + do periodic update in a background run
3. Can we build it incrementally (without first run)? Also yes

What are the operation? What are the costs? What tech stack should be used?

• update to one task can trigger updates to a few other tasks and a doc (intensive)
• select in admin panel (not very intensive and from widgets, which can be cached)

Storage:

• tasks will have two progresses: own & total (with children)
• docs will have one progress

Optimization strategies:

• [ ] pass increase in percentage to update_total_progress instead of doing some selects in it
• [ ] compute total_progress periodically in a background task

[KrzysztofMadejski]

Current situation is that on each entry total_progress of task and its parents is recomputed during request cycle.

Presentation in the UI: