m3dev / gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.
https://gokart.readthedocs.io/en/latest/
MIT License
305 stars 57 forks source link

Feature Request: load_dumps_of_requirements #214

Closed Hi-king closed 3 years ago

Hi-king commented 3 years ago

When I debug TaskOnKart, I often download dumps of requires. But it's a little bit time consuming to search which dump is actual one.

Assuming below class structure, we need to find actual dumps of TaskFoo and TaskBar to reproduce bugs on local. But it's hard to find exact dump of TaskFoo from TaskFoo_dkajflksajflasjd, TaskFoo_kfkajfklsajljfalkfa, TaskFoo_gjioaquijoree ...

class TaskA:
   a = Parameter()
   def requires():
        return dict(
             foo=TaskFoo(),
             bar=TaskBar(a=self.a)
        )

Below API might help us rapid debugging of TaskOnKart

def load_dumps_of_requirements(class, task_params_path) -> dict[str, Object]

$ load_dumps_of_requirements(TaskA, 'gs://foo/logs/task_params/TaskA_jfdafj;sajfsa.pkl')
-> {
  'foo': DataFrame(...),
  'bar': DataFrame(...)
}
Hi-king commented 3 years ago

I believe #241 have solved this. Thx @mski-iksm