python-bonobo / bonobo

Extract Transform Load for Python 3.5+
https://www.bonobo-project.org/
Apache License 2.0
1.59k stars 146 forks source link

Set variables in graph execution context #248

Open hartym opened 6 years ago

hartym commented 6 years ago

Global flags (as in, global vars) are not recommended, because it makes the code less reuseable. There is no "simple" api for this, although it's doable using the execution context (basically, a namespace that only lives the lifetime of an execution)

As you ask, there are api in "NodeExecutionContext" to do so, but not in "GraphExecutionContext", which is the one returned by bonobo.run(...). So I'm a bit reluctant to give you a recipe for that ... Still


@use_context
def tr(context, ...):
    try: 
        my_errors = getattr(context.parent, 'my_errors')
    except AttributeError:
        setattr(context.parent, 'my_errors', 0)

    ...

    if error:
        context.parent.my_errors += 1

...

c = bonobo.run(...)

print(c.my_errors)

You can also use UnrecoverrableError but this is a bit like hitting CTRL-C, it will stop as computerly fast as possible, and that's not the wanted behaviour you describe. [15 h 50] Ideally, there would be a context.setdefault('my_errors', global=True) that does the same, good enhancement for 0.7 :sourire:

hartym commented 6 years ago

This is probably thread-unsafe, working on it.