m3dev / gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.
https://gokart.readthedocs.io/en/latest/
MIT License
318 stars 57 forks source link

RuntimeError: Unfulfilled dependency at run time #224

Open e-mon opened 3 years ago

e-mon commented 3 years ago

Luigi (also gokart) implicitly assumes that the file is output when the Task run is executed, and if it is not, an error Unfulfilled dependency at run time is raised like following sample code.

This behavior is a bit confusing, so how about making sure that the file is output when the run completes?

import gokart
import luigi

class TaskA(gokart.TaskOnKart):
    def run(self):
        pass

class TaskB(gokart.TaskOnKart):
    def requires(self):
        return TaskA()

    def run(self):
        pass

if __name__ == '__main__':
    gokart.run(['TaskB', '--local-scheduler'], set_retcode=True)

error:

Traceback (most recent call last):
  File "/Users/e-mon/.pyenv/versions/3.7.5/lib/python3.7/site-packages/luigi/worker.py", line 176, in run
    raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: TaskA__99914b932b
vaaaaanquish commented 3 years ago

I think the best way to do this is to have the ability to support in luigi. https://github.com/spotify/luigi/issues/3095