Disallow fork in a subinterpreter affects multiprocessing plugin

[9ddeca9f-a5da-4419-a818-48482f8f6dcd]

BPO	43514
Nosy	@tiran, @franku

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['expert-C-API', 'type-bug', '3.8'] title = 'Disallow fork in a subinterpreter affects multiprocessing plugin' updated_at = user = 'https://github.com/franku' ``` bugs.python.org fields: ```python activity = actor = 'franku' assignee = 'none' closed = False closed_date = None closer = None components = ['C API'] creation = creator = 'franku' dependencies = [] files = [] hgrepos = [] issue_num = 43514 keywords = [] message_count = 5.0 messages = ['388841', '388842', '388876', '388877', '388879'] nosy_count = 2.0 nosy_names = ['christian.heimes', 'franku'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue43514' versions = ['Python 3.8'] ```

[9ddeca9f-a5da-4419-a818-48482f8f6dcd]

Related to this issue #78832, our Bareos libcloud plugin cannot be run with Python > 3.7. We are using subprocesses in a C-subinterpreter environment.

Is there a way to circumvent rewriting our code completely?

[tiran]

Could you please post the error message and either post a minimal example or give us a link to your code?

[9ddeca9f-a5da-4419-a818-48482f8f6dcd]

Here is part of the gdb backtrace:

[franku@franku py3plug-fd-libcloud]$ sbin/bareos_fd-py3plug-fd-libcloud -f -c etc/bareos
Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter Python runtime state: initialized

Current thread 0x00007f0dc2ab7640 (most recent call first): File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 66 in _launch File "/usr/lib64/python3.9/multiprocessing/popenfork.py", line 19 in \_init File "/usr/lib64/python3.9/multiprocessing/context.py", line 277 in _Popen File "/usr/lib64/python3.9/multiprocessing/context.py", line 224 in _Popen File "/usr/lib64/python3.9/multiprocessing/process.py", line 121 in start File "/home/franku/git/bareos/master/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py", line 102 in __init File "/home/franku/git/bareos/master/core/src/plugins/filed/python/libcloud/BareosFdPluginLibcloud.py", line 267 in start_backup_job File "/home/franku/git/bareos/master/core/src/plugins/filed/python/pyfiles/BareosFdPluginBaseclass.py", line 285 in handle_plugin_event File "/home/franku/git/bareos/master/core/src/plugins/filed/python/pyfiles/BareosFdWrapper.py", line 38 in handle_plugin_event Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter Python runtime state: initialized

[9ddeca9f-a5da-4419-a818-48482f8f6dcd]

Initialization of the Python interpreter in C-code is here: https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/python-fd.cc#L189

The actual Python code that starts the child processes is here: https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py#L110

Where this is one of the actual Processes: https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/bareos_libcloud_api/bucket_explorer.py#L45

Inherited from this base class: https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/bareos_libcloud_api/process_base.py#L36

[9ddeca9f-a5da-4419-a818-48482f8f6dcd]

These lines correspond (due to dirty working copy):

File "/home/franku/git/bareos/master/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py", line 102 in __init__

The actual Python code that starts the child processes is here: https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py#L110

[terryjreedy]

Closing because the failure caused by #78832 is intentional, not a bug; the question "Is there a way to circumvent rewriting our code completely?" belongs elsewhere Discuss-help; and because a mannequin user would not get an answer given here.