search

AustralianSynchrotron / lightflow

A lightweight, distributed workflow system
https://australiansynchrotron.github.io/lightflow/
BSD 3-Clause "New" or "Revised" License
101 stars 18 forks source link

How to import custom modules in workflow? #9

Open ghostviper opened 6 years ago

ghostviper commented 6 years ago

When I import the custom module in the workflow callback function, the execution goes wrong.

[12/01/2018 13:21:32][ERROR] ForkPoolWorker-7 | Task lightflow.queue.jobs.execute_task[6cd45833-e7dd-4753-abe2-3b5a1b72886f] raised unexpected: ImportError("No module named 'lib'",) Traceback (most recent call last): File "/usr/local/python3/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task R = retval = fun(*args, *kwargs) File "/usr/local/python3/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__ return self.run(args, kwargs) File "/usr/local/python3/lib/python3.5/site-packages/lightflow/queue/jobs.py", line 214, in execute_task event_type=JobEventName.Aborted)) File "/usr/local/python3/lib/python3.5/site-packages/lightflow/models/task.py", line 245, in _run result = self.run(data, store, signal, context) File "/usr/local/python3/lib/python3.5/site-packages/lightflow/tasks/python_task.py", line 64, in run result = self._callback(data, store, signal, context, kwargs) File "./tasks/filter_task.py", line 14, in inc_number from lib.filter import HtmlFilter ImportError: No module named 'lib'

portrain commented 6 years ago

Sorry for the late reply. Could you please add the path to your custom library to the libraries list in the configuration file for your worker? This will make sure the path to your custom library is being added to the PYTHONPATH. Does this solve your problem?