Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.71k
stars
2.39k
forks
source link
Sequential processing of a specific task with resouces:1 seems to be not working #3299
maybe I am doing something wrong but I have the following situation: I have one specific task named PrefetchTask which can not run in parallel. To achieve this I have introduced a resource like resources = {'prefetch': 1} in the class PrefetchTask. In the luigi configuration at /etc/luigi/luigi.cfg I have added the following section:
[resources]
prefetch=1
If I then submit two tasks, the first one is status running the second one is status pending because of 1 was not granted run persmission by the scheduler. The second task is never run and after some time it is in status done without creating the output file. I was expecting it to be scheduled after task one is finished and the run. Is it working as intented?
Thanks for your help,
Joshy
PS: Is there a better way to achieve what I want?
PPS: Running luigi version: 3.5.1 with python 3.11.8
Hi
maybe I am doing something wrong but I have the following situation: I have one specific task named
PrefetchTaskwhich can not run in parallel. To achieve this I have introduced a resource likeresources = {'prefetch': 1}in the classPrefetchTask. In the luigi configuration at/etc/luigi/luigi.cfgI have added the following section:If I then submit two tasks, the first one is status
runningthe second one is statuspendingbecause of1 was not granted run persmission by the scheduler. The second task is never run and after some time it is in statusdonewithout creating the output file. I was expecting it to be scheduled after task one is finished and the run. Is it working as intented? Thanks for your help, Joshy PS: Is there a better way to achieve what I want? PPS: Running luigi version: 3.5.1 with python 3.11.8