botify-labs / simpleflow

Python library for dataflow programming.
https://botify-labs.github.com/simpleflow/
MIT License
68 stars 24 forks source link

Add middleware #380

Closed Agyar closed 3 years ago

Agyar commented 3 years ago

This PR adds the loading of a middleware to a simpleflow worker. For simplicity this middleware is then passed to each activity task, effectively allowing us to handle both metrology activities and simple funcs with minimal headaches.

Leaving the test for another PR as it does not pass on travis for some reason

Agyar commented 3 years ago

Great! Nitpicks here and there muscle slightly_smiling_face

I'm not 100% sure we should ignore middleware loading errors; could you expand on why you did this?

Fully agree on not being sure Why I choose to pass here: it's on the ActivityTask, and arguably I don't like failing at that point

What seems a reasonable idea would be to pass the object loaded from above (say load it from the worker) but failing from the worker seems like a bad idea too ? What do you think

ybastide commented 3 years ago

What seems a reasonable idea would be to pass the object loaded from above (say load it from the worker) but failing from the worker seems like a bad idea too ? What do you think

Maybe just log the errors?

diff --git a/simpleflow/task.py b/simpleflow/task.py
index e34e3069..2989c40c 100644
--- a/simpleflow/task.py
+++ b/simpleflow/task.py
@@ -13,7 +13,7 @@ from simpleflow.base import Submittable
 from simpleflow.history import History
 from simpleflow.utils import import_from_module

-from . import futures
+from . import futures, logger
 from .activity import Activity

 if TYPE_CHECKING:
@@ -90,7 +90,7 @@ class ActivityTask(Task):
             try:
                 func = import_from_module(pre)
             except AttributeError:
-                pass
+                logger.exception("Cannot import a pre middleware")
             else:
                 self.pre_execute_funcs.append(func)

@@ -98,7 +98,7 @@ class ActivityTask(Task):
             try:
                 func = import_from_module(post)
             except AttributeError:
-                pass
+                logger.exception("Cannot import a post middleware")
             else:
                 self.post_execute_funcs.append(func)