search

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.89k stars 2.4k forks source link

Many pending tasks, but no worker will process them. #1215

Closed leafjungle closed 9 years ago

leafjungle commented 9 years ago

hii, guys.

I am sure that all the pending tasks are not "disabled" (retry times=100, and they are processed 0 or 1 times only).

what is the reason?

Tarrasch commented 9 years ago

Many pending tasks, but no worker will process them.

Have you seen http://tarrasch.github.io/luigid-basics-jun-2015/#/? It might give more insights of how luigi works. Also check out the assistant feature.

leafjungle commented 9 years ago

@Tarrasch, I see that doc, is it means that I have to start task it again (by crontab?) to finish the pending tasks?

Why does not the schduler automatically start a worker to check and do the pending task?

leafjungle commented 9 years ago

worker-keep-alive: True

I have config it in the client.cfg

leafjungle commented 9 years ago

In my design, each time I start a task A, a shell script will first check "ps -ef |grep A", if found a procedure, it will quit. otherwise start task A(N+1).

But I found that even task A is not finished, the procedure is gone.

leafjungle commented 9 years ago

I see in the log:

luigi.rpc.RPCError: Errors (1 attempts) when connecting to remote scheduler 'localhost'

erikbern commented 9 years ago

Closing this since not enough background