Seg Fault on OSX when multiprocessing

[afda17ab-fbfc-4830-92ff-55f81b26fe15]

BPO	37677
Nosy	@ronaldoussoren, @pitrou, @ned-deily, @applio, @tirkarthi, @tstonis

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['OS-mac', 'type-crash'] title = 'Seg Fault on OSX when multiprocessing' updated_at = user = 'https://bugs.python.org/alvis' ``` bugs.python.org fields: ```python activity = actor = 'ronaldoussoren' assignee = 'none' closed = True closed_date = closer = 'ronaldoussoren' components = ['macOS'] creation = creator = 'alvis' dependencies = [] files = [] hgrepos = [] issue_num = 37677 keywords = [] message_count = 6.0 messages = ['348429', '348430', '349514', '354243', '354325', '378995'] nosy_count = 7.0 nosy_names = ['ronaldoussoren', 'pitrou', 'ned.deily', 'davin', 'xtreak', 'alvis', 'timstonis'] pr_nums = [] priority = 'normal' resolution = 'out of date' stage = 'resolved' status = 'closed' superseder = None type = 'crash' url = 'https://bugs.python.org/issue37677' versions = ['Python 2.7', 'Python 3.6'] ```

[afda17ab-fbfc-4830-92ff-55f81b26fe15]

Encountered a crash with the following logs. I have already added no_proxy=*, and it fixed the issue which i can replicate consistent. However, this crash does not happen every time, maybe once in a few week.

Process: Python [66435] Path: /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 3.6.8 (3.6.8) Code Type: X86-64 (Native) Parent Process: Python [65077] Responsible: Python [66435] User ID: 501

Date/Time: 2019-07-24 21:45:29.722 +0800 OS Version: Mac OS X 10.14.2 (18C54) Report Version: 12 Bridge OS Version: 3.5 (16P5125) Anonymous UUID: 3F729BF3-2BB1-3830-B6FE-D22CA431DE5A

Time Awake Since Boot: 1600000 seconds

System Integrity Protection: enabled

Crashed Thread: 2

Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000109b23aa2 Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Segmentation fault: 11 Termination Reason: Namespace SIGNAL, Code 0xb Terminating Process: exc handler [66435]

VM Regions Near 0x109b23aa2: MALLOCLARGE 0000000109aa3000-0000000109b23000 [ 512K] rw-/rwx SM=COW
--> \_TEXT 0000000109b27000-0000000109b3d000 [ 88K] r-x/rwx SM=COW /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/.dylibs/libgcc_s.1.dylib

Application Specific Information: multi-threaded process forked crashed on child side of fork pre-exec

Thread 2 Crashed: 0 libsystem_trace.dylib 0x00007fff7d6fdab3 _os_log_cmp_key + 4 1 libsystem_c.dylib 0x00007fff7d5aaa5e rb_tree_find_node + 53 2 libsystem_trace.dylib 0x00007fff7d6fbc81 os_log_create + 370 3 libnetwork.dylib 0x00007fff7c04058f nwlog_legacy_init_once + 47 4 libsystempthread.dylib 0x00007fff7d6e1ce4 \_pthread_once_handler + 66 5 libsystem_platform.dylib 0x00007fff7d6d7acb _os_once_callout + 18 6 libsystem_pthread.dylib 0x00007fff7d6e1c7f pthread_once + 56 7 libnetwork.dylib 0x00007fff7c024ccd networkd_settings_init + 669 8 libnetwork.dylib 0x00007fff7c024982 networkd_settings_get_bool + 18 9 libnetwork.dylib 0x00007fff7c0244f0 -[NWConcrete_nw_parameters initWithStack:] + 160 10 libnetwork.dylib 0x00007fff7c023d97 nw_path_create_evaluator_for_endpoint + 487 11 libnetwork.dylib 0x00007fff7c2451bc nw_nat64_v4_address_requires_synthesis + 220 12 libsystem_info.dylib 0x00007fff7d609e83 _gai_nat64_v4_address_requires_synthesis + 67 13 libsystem_info.dylib 0x00007fff7d609210 _gai_nat64_second_pass + 512 14 libsystem_info.dylib 0x00007fff7d5e7b4a si_addrinfo + 1978 15 libsystem_info.dylib 0x00007fff7d5e7262 _getaddrinfo_internal + 242 16 libsystem_info.dylib 0x00007fff7d5e715d getaddrinfo + 61 17 _socket.cpython-36m-darwin.so 0x00000001092c24ed socket_getaddrinfo + 621 18 org.python.python 0x0000000106d63697 _PyCFunction_FastCallDict + 183 19 org.python.python 0x0000000106de5747 call_function + 439 20 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 21 org.python.python 0x0000000106de61af _PyEval_EvalCodeWithName + 2447 22 org.python.python 0x0000000106de6ae1 fast_function + 545 23 org.python.python 0x0000000106de5721 call_function + 401 24 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 25 org.python.python 0x0000000106de61af _PyEval_EvalCodeWithName + 2447 26 org.python.python 0x0000000106de6ae1 fast_function + 545 27 org.python.python 0x0000000106de5721 call_function + 401 28 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 29 org.python.python 0x0000000106de6a3d fast_function + 381 30 org.python.python 0x0000000106de5721 call_function + 401 31 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 32 org.python.python 0x0000000106de61af _PyEval_EvalCodeWithName + 2447 33 org.python.python 0x0000000106de6ae1 fast_function + 545 34 org.python.python 0x0000000106de5721 call_function + 401 35 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 36 org.python.python 0x0000000106de61af _PyEval_EvalCodeWithName + 2447 37 org.python.python 0x0000000106de6dfb _PyFunction_FastCallDict + 763 38 org.python.python 0x0000000106d1a6b7 _PyObject_FastCallDict + 247 39 org.python.python 0x0000000106d1a7d5 _PyObject_Call_Prepend + 149 40 org.python.python 0x0000000106d1a4f0 PyObject_Call + 96 41 org.python.python 0x0000000106d7badd slot_tp_init + 125 42 org.python.python 0x0000000106d77d09 type_call + 313 43 org.python.python 0x0000000106d1a685 _PyObject_FastCallDict + 197 44 org.python.python 0x0000000106de5648 call_function + 184 45 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 46 org.python.python 0x0000000106de6a3d fast_function + 381 47 org.python.python 0x0000000106de5721 call_function + 401 48 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 49 org.python.python 0x0000000106de6a3d fast_function + 381 50 org.python.python 0x0000000106de5721 call_function + 401 51 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 52 org.python.python 0x0000000106de6a3d fast_function + 381 53 org.python.python 0x0000000106de5721 call_function + 401 54 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 55 org.python.python 0x0000000106de6a3d fast_function + 381 56 org.python.python 0x0000000106de5721 call_function + 401 57 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 58 org.python.python 0x0000000106de6a3d fast_function + 381 59 org.python.python 0x0000000106de5721 call_function + 401 60 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 61 org.python.python 0x0000000106de6a3d fast_function + 381 62 org.python.python 0x0000000106de5721 call_function + 401 63 org.python.python 0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559 64 org.python.python 0x0000000106de6c5c _PyFunction_FastCallDict + 348 65 org.python.python 0x0000000106d1a6b7 _PyObject_FastCallDict + 247 66 org.python.python 0x0000000106d1a7d5 _PyObject_Call_Prepend + 149 67 org.python.python 0x0000000106d1a4f0 PyObject_Call + 96 68 org.python.python 0x0000000106e2d396 t_bootstrap + 70 69 org.python.python 0x0000000106e26be9 pythread_wrapper + 25 70 libsystem_pthread.dylib 0x00007fff7d6e3305 _pthread_body + 126 71 libsystem_pthread.dylib 0x00007fff7d6e626f _pthread_start + 70 72 libsystem_pthread.dylib 0x00007fff7d6e2415 thread_start + 13

[tirkarthi]

Thanks for the report. It would be helpful to have a standalone reproducer for this issue to try reproducing it in 3.6 and other versions.

[afda17ab-fbfc-4830-92ff-55f81b26fe15]

Hi, sorry i am unable to provide a standalone script that cause this issue, as it happen once every few weeks. Is it similar to the no_proxy=* issue?

[8ab26e91-1a97-45ca-a347-03c04247ad51]

Hi, I have the exact same issue and same stack trace from getaddrinfo (except Python 2.7). Same as original poster, it only happens every couple of weeks and I don't know how to reproduce. For me it's in an Odoo instance that spawns multiple workers with connections to a postgres database. I think this is a MacOS bug, since the issue does not happen on High Sierra (10.13).

[ronaldoussoren]

Looking at the stack trace this is a similar issue as the "no_proxy" one, but in some sense worse because it happens when just calling a regular unix API (not an Apple framework API).

The first few lines of the stack trace are for code in the "Network" framework which is new in macOS 10.14 and is based on lib dispatch and hence uses threads. That results in the same problem as the "no_proxy" one: code that causes problems when called after os.fork().

That's a bug in macOS.

In Python 3.8 the multiprocessing module switched to "spawn" mode instead of "fork" mode because of this. You can do this manually for earlier versions (https://docs.python.org/3.7/library/multiprocessing.html#contexts-and-start-methods)

P.S. A reproducer would still be nice as that would make it easier to raise and issue with Apple about this.

[ronaldoussoren]

I'm closing this issue because this crash is caused by a platform issue, and Python 3.8 has a workaround by using a different spawn method in multiprocessing.