search

OpenConceptLab / ocl_issues

Issues for all OCL repos. NOTE: Install ZenHub Browser Extension and request access to the OCL Roadmap board to view all issues and to contribute
4 stars 2 forks source link

Persist partial progress for new bulk import #1923

Open rkorytkowski opened 2 months ago

rkorytkowski commented 2 months ago

Currently, when requesting progress for the new bulk import, we visit all tasks in the chain and calculate the progress for completed tasks. For bigger imports such as r4.core there is a lot of tasks and at some point requesting the progress is running too long and timing out.

We need to persist partial progress for the groups in the same way as we persist final results by introducing a final task in a group that calculates the results for the group and stores them as a celery result.

The partial results need to be taken into account when calculating total progress.

paynejd commented 4 days ago

@rkorytkowski Please provide update -- when will this be closed out?