mindee / tawazi

A DAG Scheduler library written in pure python
https://mindee.github.io/tawazi/
Apache License 2.0
83 stars 5 forks source link

[FEATURE] Verbose logging for node execution #38

Closed matthiasmindee closed 1 year ago

matthiasmindee commented 1 year ago

Problem Since we access the function through a decorator, and we 'patch' its execution when resolving the dag dependency, we 'execute' the decorated function earlier that expected in the code, hence if there's another decorator it messes its logic up

Suggested Solution Either find a smart way of composing decorators, or overwrite the default logging strategy (in this case). It might indicate a design flaw in the dependency resolution here so it might be worthwhile to investigate

bashirmindee commented 1 year ago

can you provide a code showing the pb ?

matthiasmindee commented 1 year ago
from tawazi import op, to_dag
from functools import wraps

def my_little_logger(func):

    @wraps(func)
    def log(*args, **kwargs):
        print("this should print before execution")
        res = func(*args, **kwargs)
        print("this should print after execution")
        return res

    return log

@my_little_logger
@op
def a():
    return 'titi'

@my_little_logger
@op
def b(a):
    return "tata" + a

@to_dag
def pipe():
    t = b(a())

if __name__ == '__main__':
    pipe().execute()

which yields:

this should print before execution
this should print after execution
this should print before execution
this should print after execution
a
b
bashirmindee commented 1 year ago

The problem here is that tawazi lost support for this function definition:

def foo(*args, **kwrags):
     pass

and because there is no tests for this case it got passed unnoticed. I will try to make a fix... and effectiveley this indicates a problem in the parameter passing phase / dependency resolution that is fundamentally wrong and needs correction by a redesign.