onnx / turnkeyml

The no-code AI toolchain
Apache License 2.0
73 stars 16 forks source link

TurnkeyML Tools seem to be incompatible with python multiprocessing #219

Closed danielholanda closed 2 months ago

danielholanda commented 2 months ago

Description

TurnkeyML ManagementTools seem to be incompatible with python multiprocessing as using it causes circular imports.

Example

Running the following sample Management Tool causes the error shown below.

import argparse
from multiprocessing import Process
from turnkeyml.tools.management_tools import ManagementTool

def print_message(message):
    print(message)

class ExamplePluginTool(ManagementTool):
    """
    Example of a Tool installed by a plugin. Note that this docstring appears
    in the help menu when `turnkey example-plugin-tool -h` is called.
    """

    unique_name = "example-plugin-tool"

    def __init__(self):
        super().__init__()

    @staticmethod
    def parser(add_help: bool = True) -> argparse.ArgumentParser:
        parser = __class__.helpful_parser(
            short_description="This is an examples tool from the Tool Plugins example",
            add_help=add_help,
        )

        return parser

    def run(self, cache_dir: str):

        # Run using multiprocessing
        message = f"Running using multiprocessing. Cache dir is {cache_dir}"
        process = Process(target=print_message, args=(message,))
        process.start()
        process.join()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\danie\miniconda3\envs\digest310\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\danie\miniconda3\envs\digest310\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "c:\users\danie\turnkeyml\examples\cli\plugins\example_tool\turnkeyml_plugin_example_tool\__init__.py", line 1, in <module>
    from .tool import ExamplePluginTool
  File "c:\users\danie\turnkeyml\examples\cli\plugins\example_tool\turnkeyml_plugin_example_tool\tool.py", line 16, in <module>
    from turnkeyml.tools.management_tools import ManagementTool
  File "C:\Users\danie\miniconda3\envs\digest310\lib\site-packages\turnkeyml\__init__.py", line 4, in <module>
    from .cli.cli import main as turnkeycli
  File "C:\Users\danie\miniconda3\envs\digest310\lib\site-packages\turnkeyml\cli\cli.py", line 9, in <module>
    from turnkeyml.sequence.tool_plugins import SUPPORTED_TOOLS
  File "C:\Users\danie\miniconda3\envs\digest310\lib\site-packages\turnkeyml\sequence\tool_plugins.py", line 29, in <module>
    if "tools" in module.implements.keys():
AttributeError: partially initialized module 'turnkeyml_plugin_example_tool' has no attribute 'implements' (most likely due to a circular import)

Please note that running the same logic from a standalone script (outside of tkml) works as intended.

Reproducing

A full example has been added to the branch dholanda/circular_import. To reproduce the issue, simply checkout that branch, install examples\cli\plugins\example_tool and run turnkey example-plugin-tool.