lisad / phaser

The missing layer for complex data batch integration pipelines
MIT License
9 stars 1 forks source link

Testing batch step more: fixed how it works when the batch step forgets to declare context param #90

Closed lisad closed 8 months ago

lisad commented 8 months ago

When the step doesn't declare context:

@batch_step
def my_simple_step(batch):
    ... 
   return batch

Then when the step is called in phase when its going through steps:

for step_function in steps:
   ...
   step_function(batch, context)

It doesn't work - it doesn't even go into the function, I think. This seems like something that might bite a developer pretty easily - they forget to add the 'context' param, which they don't even want, and can't figure out why their step isn't even running.

I found out that with the python 'inspect' library one can look at the function's signature and see if it expects a context, and only then pass the context.

jeffkole commented 8 months ago

This is one that has tripped me up several times and is always annoying to find a test failure. Thank you!