search

stephenpascoe / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
1 stars 1 forks source link

Not all Pending tasks shown #1

Closed stephenpascoe closed 9 years ago

stephenpascoe commented 9 years ago

From https://github.com/spotify/luigi/pull/1086

First, I have a couple thousand pending tasks but only 367 show up as pending on the main page. The worker page shows more than that. Did the internal representation change in a way that requires clearing the pickle?

Are any of these pending tasks upstream failures or disabled? I'm counting these independently. Maybe this is also related to the server limit (see below)?

stephenpascoe commented 9 years ago

That probably explains it, as all the other counts seemed accurate and jobs had been building up yesterday due to some overnight failures. The previous version placed such tasks both in upstream failed/disabled and pending. It might be confusing to users to change it, but I can see why you did. I think I can get used to either solution. Are upstream failed and upstream disabled all pending?

All upstream failed/disabled are pending. This seems logical to me as a way to not count pending tasks twice. I've confirmed the counts seem to add up when we have pending and failed tasks.