Report `multiprocessing` issues in `plugin_host-3.8` upstream

[willangley]

Sublime Text's plugin_host-3.8 offers the multiprocessing API, but I couldn't get it to work on macOS! I tried multiple strategies:

• forking plugin_host-3.8
• spawn-ing plugin_host-3.8
• setting the executable to python3 and spawn-ing it

and they all failed; attempts to communicate with the child process would report broken pipe and investigating it in the Console showed it had exited immediately.

Report these issues upstream. It likely either (a) needs work, and (b) needs to be made unavailable until it works reliably.

[willangley]

plugin_host-3.8

Playing with plugin_host-3.8 makes it pretty clear why spawning it doesn't help: it expects to talk to a parent Sublime Text instance and dies if it can't.

% "/Applications/Sublime Text.app/Contents/MacOS/plugin_host-3.8"
Unexpected number of arguments, expected 6
% pgrep plugin_host-3.8 | xargs -n 1 ps -fp
  UID   PID  PPID   C STIME   TTY           TIME CMD
  501 18708 18702   0  5:57PM ??         0:01.29 /Applications/Sublime Text.app/Contents/MacOS/plugin_host-3.8 18702 /Applications/Sublime Text.app/Contents/MacOS/Sublime Text /Users/wmangley3/Library/Application Support/Sublime Text 3 /Users/wmangley3/Library/Application Support/Sublime Text 3 /Users/wmangley3/Library/Application Support/Sublime Text 3/Packages
% /Applications/Sublime\ Text.app/Contents/MacOS/plugin_host-3.8 18702 /Applications/Sublime\ Text.app/Contents/MacOS/Sublime\ Text /Users/wmangley3/Library/Application\ Support/Sublime\ Text 3 /Users/wmangley3/Library/Application\ Support/Sublime\ Text 3 /Users/wmangley3/Library/Application\ Support/Sublime\ Text 3/Packages
unable to open channels
%

This would have saved me significant time if I'd tried it then, too. Oops!

[willangley]

python3

This doesn't work with multiprocessing.set_executable('python3'), but reading the docs again shows this method being called with full paths; it apparently does not do path resolution.

multiprocessing actually seems to work on a toy example with multiprocessing.set_executable('/usr/local/bin/python3'). I'm now curious if it will work for this after all

I used a technique I hadn't before while trying this:

• launching under a debugger

which was able to show exceptions when processes failed to launch! Funnily enough, the remaining issue (once I fixed the executable path) was trying to launch a function typed into the interpreter under multiprocessing, which may not work.

Creating a toy plugin with the function worked though!

#!/usr/bin/env python3

def foo():
    print('hello')

if __name__ == '__main__':
    foo()

[willangley]

python3, part 2

I wasn't able to use a multiprocessing queue, though; as soon as I changed the plugin to use a queue,

#!/usr/bin/env python3

def foo(q):
    q.put('Hello world!')

if __name__ == '__main__':
    foo()

process launch failed with:

Traceback (most recent call last):
  File "<frozen zipimport>", line 520, in _get_decompress_func
ModuleNotFoundError: No module named 'zlib'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen zipimport>", line 568, in _get_data
  File "<frozen zipimport>", line 523, in _get_decompress_func
zipimport.ZipImportError: can't decompress data; zlib not available

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/queues.py", line 21, in <module>
    from queue import Empty, Full
  File "<frozen zipimport>", line 241, in load_module
  File "<frozen zipimport>", line 709, in _get_module_code
  File "<frozen zipimport>", line 570, in _get_data
zipimport.ZipImportError: can't decompress data; zlib not available

I think for a second, and remember the _Popen() internal state that I observed in the debugger:

{
    'self': <multiprocessing.popen_spawn_posix.Popen object at 0x105703310>,
    'process_obj': <Process name='Process-1' parent=19780 initial>,
    'resource_tracker': <module 'multiprocessing.resource_tracker' from '/Applications/Sublime Text.app/Contents/MacOS/Lib/python3.8.zip/multiprocessing/resource_tracker.pyc'>,
    'tracker_fd': 46,
    'prep_data': {
        'log_to_stderr': False,
        'name': 'Process-1',
        'sys_path': ['/Applications/Sublime Text.app/Contents/MacOS/Lib/python3.8.zip', '/Applications/Sublime Text.app/Contents/MacOS/Lib/python38', '/Applications/Sublime Text.app/Contents/MacOS/Lib/python3', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Lib/python38', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Packages', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Packages/PreviewInMarked/third_party'],
        'sys_argv': [''],
        'orig_dir': '/Applications/Sublime Text.app/Contents/MacOS',
        'dir': '/Applications/Sublime Text.app/Contents/MacOS',
        'start_method': 'spawn'
    },
    'fp': <_io.BytesIO object at 0x105771db0>
    'parent_r': 44,
    'child_w': 47,
    'child_r': 48,
    'parent_w': 49,
    'cmd': ['/usr/local/bin/python3', '-O', '-S', '-E', '-s', '-c', 'from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=46, pipe_handle=48)', '--multiprocessing-fork']
}

I don't think that contains zlib… sure enough, it doesn't. The system Pythons on my machine have an extra folder in sys.path that this doesn't, presumably because it's linked statically with plugin_host-3.8 and thus not needed,

['', '/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages', '/usr/local/Cellar/pybind11/2.6.2/libexec/lib/python3.9/site-packages']

[willangley]

python3, part 3

This can be worked around by setting sys.path before launching a process:

>>> sys.path
['/Applications/Sublime Text.app/Contents/MacOS/Lib/python3.8.zip', '/Applications/Sublime Text.app/Contents/MacOS/Lib/python38', '/Applications/Sublime Text.app/Contents/MacOS/Lib/python3', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Lib/python38', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Packages', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Packages/PreviewInMarked/third_party']
>>> sys.path = ['/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages', '/usr/local/Cellar/pybind11/2.6.2/libexec/lib/python3.9/site-packages', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Packages', '/Users/wmangley3/Library/Application Support/Sublime Text 3/Packages/PreviewInMarked/third_party']
>>> p = multiprocessing.Process(target=PreviewInMarked.test.foo, args=(q,))
>>> pdb.Pdb(stdout=sys.__stdout__).run('p.start()')
>>> q.get_nowait()
'Hello world!'