search

ow2-proactive / scheduling

Multi-platform Scheduling and Workflows Engine
http://www.activeeon.com/workflows-scheduling
GNU Affero General Public License v3.0
62 stars 54 forks source link

CRON loops are not usable #2279

Open lpellegr opened 9 years ago

lpellegr commented 9 years ago

CRON loops are really useful to execute recurring actions. Unfortunately, some issues are encountered when the number of tasks generated by a CRON loop for a given job becomes too high. Indeed, the scheduler was not designed for handling a lot of tasks per Job.

This point should be improved since CRON loops have been used or requested for several use cases (Homeoffice, Enterome, etc.).

The first solution is to add support for a big number of tasks per job in the scheduler (related to #1982). It implies to support pagination for tasks in the REST API but also in the scheduler web portal.

Another solution that could be investigated is to see a CRON loop as a set of tasks whose each iteration triggers the execution of the tasks in a dedicated job. The new job that is related to the previous iteration could simply keep the relation to the previous job by adding the information in its name. However, a more robust and reusable solution would be to keep track of the job ID associated to the previous iteration but also the ID of the original Job (the one that triggered the first iteration) at the DB level.

tobwiens commented 8 years ago

This is probably fixed by #2357, we can close this?!

lpellegr commented 8 years ago

I am not sure it is fixed yet, but this is only my opinion. Pagination is available for tasks from clients but internally (scheduler side) all tasks are still fetched in memory whatever the number of tasks contained by a job?

I think there is still some remaining tasks as pointed by the story on Trello. Maybe a new story could be created to address these points.

lpellegr commented 8 years ago

Besides, I was wondering if some performance tests have been written to check the feature? I think CRON features should be checked with at least 3 jobs each having 1576800 or more tasks. This last value depicts a workflow with 3 tasks that are executed for a year each minute.

paraita commented 8 years ago

@lpellegr is right, all tasks (for a given job) are still loaded in-memory (server-side). Pagination has only been implemented client-side (REST,REST-CLI and portal). The job is quite big (no pun intended) and is being worked on at the moment. But yeah, a story should help to keep track of all the scattered work.

tobwiens commented 8 years ago

@lpellegr @paraita Okay let's make this an an Epic. And I will move it to 7.x.x.

fviale commented 1 year ago

This issue is still present. Pagination of tasks in the scheduler memory is a big challenge due to the dynamicity of workflows.

For example, when a loop is initiated, it needs to replicate a whole block of tasks (which must be in memory) by walking the workflow graph.

Accordingly, this improvement, even though desired, is very complex.

mbenguig commented 11 months ago

@fviale Task pagination is now supported by ProActive. Can we close this issue ?