m3dev / gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.
https://gokart.readthedocs.io/en/latest/
MIT License
318 stars 57 forks source link

[bug] ListTaskInstanceParameter does not move when not specific `requires` method #195

Closed vaaaaanquish closed 3 years ago

vaaaaanquish commented 3 years ago
import gokart

class Piyo(gokart.TaskOnKart):
    def run(self):
        self.dump('piyo')

class Hoge(gokart.TaskOnKart):
    hoge = gokart.ListTaskInstanceParameter()

    def run(self):
        hoge = self.load_data_frame('hoge')

gokart.build(Hoge(hoge=[Piyo(), Piyo()]))
Traceback (most recent call last):
  File "python3.7/site-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "python3.7/site-packages/luigi/worker.py", line 133, in _run_get_new_deps
    task_gen = self.task.run()
  File "hoge.py", line 17, in run
    df = self.load_data_frame('hoge')
  File "python3.7/site-packages/gokart/task.py", line 236, in load_data_frame
    dfs = self.load(target=target)
  File "python3.7/site-packages/gokart/task.py", line 211, in load
    return _load(self._get_input_targets(target))
  File "python3.7/site-packages/gokart/task.py", line 276, in _get_input_targets
    return self.input()[target]
TypeError: list indices must be integers or slices, not str
vaaaaanquish commented 3 years ago
  File "python3.7/site-packages/gokart/task.py", line 276, in _get_input_targets
    return self.input()[target]
vaaaaanquish commented 3 years ago

Not working.

class Hoge(gokart.TaskOnKart):
    hoge = gokart.ListTaskInstanceParameter()

    def run(self):
        hoge = self.load_data_frame('hoge')

Working:+1:

class Hoge(gokart.TaskOnKart):
    hoge = gokart.ListTaskInstanceParameter()

    def requires(self):
        return {'hoge': self.hoge}

    def run(self):
        hoge = self.load_data_frame('hoge')
vaaaaanquish commented 3 years ago

We must check List[TaskOnKart] here

https://github.com/m3dev/gokart/blob/master/gokart/task.py#L79

hirosassa commented 3 years ago

Great work @vaaaaanquish for your detailed investigation and finding how to reproduce!