TurnkeyML ManagementTools seem to be incompatible with python multiprocessing as using it causes circular imports.
Example
Running the following sample Management Tool causes the error shown below.
import argparse
from multiprocessing import Process
from turnkeyml.tools.management_tools import ManagementTool
def print_message(message):
print(message)
class ExamplePluginTool(ManagementTool):
"""
Example of a Tool installed by a plugin. Note that this docstring appears
in the help menu when `turnkey example-plugin-tool -h` is called.
"""
unique_name = "example-plugin-tool"
def __init__(self):
super().__init__()
@staticmethod
def parser(add_help: bool = True) -> argparse.ArgumentParser:
parser = __class__.helpful_parser(
short_description="This is an examples tool from the Tool Plugins example",
add_help=add_help,
)
return parser
def run(self, cache_dir: str):
# Run using multiprocessing
message = f"Running using multiprocessing. Cache dir is {cache_dir}"
process = Process(target=print_message, args=(message,))
process.start()
process.join()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\danie\miniconda3\envs\digest310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\danie\miniconda3\envs\digest310\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "c:\users\danie\turnkeyml\examples\cli\plugins\example_tool\turnkeyml_plugin_example_tool\__init__.py", line 1, in <module>
from .tool import ExamplePluginTool
File "c:\users\danie\turnkeyml\examples\cli\plugins\example_tool\turnkeyml_plugin_example_tool\tool.py", line 16, in <module>
from turnkeyml.tools.management_tools import ManagementTool
File "C:\Users\danie\miniconda3\envs\digest310\lib\site-packages\turnkeyml\__init__.py", line 4, in <module>
from .cli.cli import main as turnkeycli
File "C:\Users\danie\miniconda3\envs\digest310\lib\site-packages\turnkeyml\cli\cli.py", line 9, in <module>
from turnkeyml.sequence.tool_plugins import SUPPORTED_TOOLS
File "C:\Users\danie\miniconda3\envs\digest310\lib\site-packages\turnkeyml\sequence\tool_plugins.py", line 29, in <module>
if "tools" in module.implements.keys():
AttributeError: partially initialized module 'turnkeyml_plugin_example_tool' has no attribute 'implements' (most likely due to a circular import)
Please note that running the same logic from a standalone script (outside of tkml) works as intended.
Reproducing
A full example has been added to the branch dholanda/circular_import. To reproduce the issue, simply checkout that branch, install examples\cli\plugins\example_tool and run turnkey example-plugin-tool.
Description
TurnkeyML ManagementTools seem to be incompatible with python multiprocessing as using it causes circular imports.
Example
Running the following sample Management Tool causes the error shown below.
Please note that running the same logic from a standalone script (outside of tkml) works as intended.
Reproducing
A full example has been added to the branch
dholanda/circular_import
. To reproduce the issue, simply checkout that branch, installexamples\cli\plugins\example_tool
and runturnkey example-plugin-tool
.