Closed kennedyjosh closed 1 year ago
Can you provide a minimal self-contained example? I tried a simple example, and I'm not seeing an issue yet.
Python 3.9.6 (default, Jul 3 2021, 08:33:50)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import pathos as pa
>>> pa.multiprocessing._ProcessPool
<class 'multiprocess.pool.Pool'>
>>>
>>> p = pa.multiprocessing._ProcessPool()
>>> p.map(lambda x: Ellipsis, [Ellipsis]*4)
[Ellipsis, Ellipsis, Ellipsis, Ellipsis]
>>> pa.__version__
'0.2.9.dev0'
>>>
Also, from your traceback, I see dill is punting to pickle, which is failing. dill
should handle the Ellipsis itself, so that's a bit odd.
>>> import dill
>>> dill.__version__
'0.3.5.dev0'
>>> dill.dumps(Ellipsis)
b'\x80\x04\x95,\x00\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\n_eval_repr\x94\x93\x94\x8c\x08Ellipsis\x94\x85\x94R\x94.'
>>> dill.loads(_)
Ellipsis
>>>
I think I might have an intuition about why it isn't working... the python module with this code is being dynamically loaded by another script, like so:
def main():
# ...
all_files = os.listdir(srcpath)
for filename in sorted(all_files):
if filename.endswith(".py"):
filepath = os.path.join(srcpath, filename)
spec = imp.spec_from_file_location(filename, filepath)
mod = imp.module_from_spec(spec)
spec.loader.exec_module(mod)
# if a module doesn't have this function, script should crash
mod.some_init_fun()
# use argparse to parse command-line args and call the appropriate function
args = parser.parser_args
args.func(args, parser)
I'm using argparse, and some_init_fun
initializes a subparser (one of which contains the problematic code from this issue). If I take the code I posted here and put it into its own file, it works fine. Here is a minimal example:
from pathos.multiprocessing import _ProcessPool as Pool
def deploy_command(region, machine_ip, branch):
print(f"called as: {region}, {machine_ip}, {branch}")
def main():
pool = Pool(processes=10)
res = pool.starmap(deploy_command,
[["us-east-1", "prod2", "master"],
["us-east-1", "prod3", "master"],
["us-east-1", "prod4", "master"]]
)
pool.close()
pool.join()
if __name__ == "__main__":
main()
Basically, when this code is called from its own file, it works. When it is called from another file as a dynamically loaded module, it doesn't work. Any ideas on how to solve this without putting everything in the same file?
Also, from your traceback, I see dill is punting to pickle, which is failing. dill should handle the Ellipsis itself, so that's a bit odd.
Yes – I noticed this. The whole reason I'm using pathos is because I had a (different) pickling error with the builtin multiprocessing library. Strange that it solved that but not this
Two things: (1) how are you dynamically loading the module from another file (code please), and (2) if you are executing anything from the shell (i.e. outside of python, in another process) are you sure that the version of python and the pythonpath is the same as in main? For example if you have multiple python installations that depends on a shell script to set the path, when you spawn a new process, you need to make sure that you are using the same python and the same path as in main.
I posted the code above, but I'll repost it here:
# dynamically load all files in srcpath and call their top-level some_init_fun method
all_files = os.listdir(srcpath)
for filename in sorted(all_files):
if filename.endswith(".py"):
filepath = os.path.join(srcpath, filename)
spec = imp.spec_from_file_location(filename, filepath)
mod = imp.module_from_spec(spec)
spec.loader.exec_module(mod)
# if a module doesn't have this function, script should crash
mod.some_init_fun()
The function some_init_fun
initializes and populates an ArgumentParser. After this for loop, the parent ArgumentParser is called to parse the command line arguments and call the correct function.
# use argparse to parse command-line args and call the appropriate function
args = parser.parser_args # parsers command-line args into variables
args.func(args, parser) # calls function from some other file
So, for the code I posted in my previous comment, the user enter some command-line command with options, and this gets passed down to the main
function which uses the Pool to call deploy_command
.
===
are you sure that the version of python and the pythonpath is the same as in main?
Good question – I inserted a print(sys.version)
and print(pathos.__version__)
right before where the error is occurring and verified that the version is the same as I posted in my first comment (Python 3.9.6, pathos 0.2.8)
Dynamically loaded modules are not guaranteed to work. What I meant with "code please" is that I can't just run your example and see if it errors or not... I'd first have to write the module you're dynamically loading, right? Where is some_init_fun
defined in your example?... and if it's not important, then maybe remove it...
I'm not able to repeat the errors you are seeing, and this issue appears a bit stale. I'm going to close this, but please feel free to reopen this if you have more input.
Code
Error & Stacktrace
Version Info
Python version: 3.9.6 pathos version: 0.2.8