Segmentation fault after request triggered from within worker on macOS

[buhrmann]

Hi, When I try to send a request to uWSGI from within one of its worker processes (while processing a previous request), I get the segmentation fault below (and it doesn't seem to depend on the details of the request). When sending the request from an external process all works fine. Not sure if that's something that should be possible? If it is, would this be a bug or am I doing it wrong?

I'm on macOS 10.13.2, in an anaconda environment with python 3.6.3, uwsgi 2.0.15, and running Flask behind uwsgi.

Here's the .ini:

[uwsgi]
file = ...
callable = APP
master = true
processes = 2
socket = 0.0.0.0:7070
protocol = http
enable-threads = true

The uwsgi launch info:

[uWSGI] getting INI configuration from whiskey.ini
*** Starting uWSGI 2.0.15 (64bit) on [Mon Jan 22 14:22:33 2018] ***
compiled with version: 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53) on 24 November 2017 13:11:34
os: Darwin-17.3.0 Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64
nodename: tb.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /Users/thomas/code/lingvo
detected binary path: /Users/thomas/anaconda/envs/teksto2/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:7070 fd 3
Python version: 3.6.3 | packaged by conda-forge | (default, Dec  9 2017, 16:20:51)  [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
Python main interpreter initialized at 0x7fd329805600
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 218280 bytes (213 KB) for 2 cores
*** Operational MODE: preforking ***
...
WSGI app 0 (mountpoint='') ready in 16 seconds on interpreter 0x7fd329805600 pid: 26544 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 26544)
spawned uWSGI worker 1 (pid: 26547, cores: 1)
spawned uWSGI worker 2 (pid: 26548, cores: 1)
*** Stats server enabled on /tmp/stats.socket fd: 15 ***

And here the trace info:

!!! uWSGI process 26548 got Segmentation Fault !!!
*** backtrace of 26548 ***
0   uwsgi                               0x0000000102b7fea0 uwsgi_backtrace + 48
1   uwsgi                               0x0000000102b803e3 uwsgi_segfault + 51
2   libsystem_platform.dylib            0x00007fff5883cf5a _sigtramp + 26
3   ???                                 0x00007fd323159d18 0x0 + 140544803446040
4   libxpc.dylib                        0x00007fff5888ddcf _xpc_connection_activate_if_needed + 248
5   libxpc.dylib                        0x00007fff5888044e xpc_connection_resume + 64
6   CoreFoundation                      0x00007fff30e201da -[_CFXPreferences _copyDaemonConnectionSettingUpIfNecessaryForRole:] + 426
7   CoreFoundation                      0x00007fff30e20267 -[_CFXPreferences withConnectionForRole:performBlock:] + 23
8   CoreFoundation                      0x00007fff30df7685 __80-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:]_block_invoke + 197
9   CoreFoundation                      0x00007fff30df74ed -[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:] + 253
10  CoreFoundation                      0x00007fff30c70f86 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 326
11  CoreFoundation                      0x00007fff30c70b03 -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 67
12  CoreFoundation                      0x00007fff30db5195 -[CFPrefsSource copyValueForKey:] + 53
13  CoreFoundation                      0x00007fff30e1ed40 __76-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:]_block_invoke + 32
14  CoreFoundation                      0x00007fff30df8cf9 __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 297
15  CoreFoundation                      0x00007fff30df8b65 -[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 341
16  CoreFoundation                      0x00007fff30e1ece3 -[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 131
17  CoreFoundation                      0x00007fff30c78253 CFPreferencesCopyAppValue + 99
18  SystemConfiguration                 0x00007fff3cfe9665 SCDynamicStoreCopyProxiesWithOptions + 155
19  _scproxy.cpython-36m-darwin.so      0x00000001060fe8b8 get_proxy_settings + 24
20  libpython3.6m.dylib                 0x00000001051bbe40 _PyCFunction_FastCallDict + 560
21  libpython3.6m.dylib                 0x000000010523f757 call_function + 439
22  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
23  libpython3.6m.dylib                 0x0000000105240e78 fast_function + 488
24  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
25  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
26  libpython3.6m.dylib                 0x0000000105240e78 fast_function + 488
27  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
28  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
29  libpython3.6m.dylib                 0x000000010524059c _PyEval_EvalCodeWithName + 3436
30  libpython3.6m.dylib                 0x0000000105240f21 fast_function + 657
31  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
32  libpython3.6m.dylib                 0x000000010523c148 _PyEval_EvalFrameDefault + 25080
33  libpython3.6m.dylib                 0x000000010524059c _PyEval_EvalCodeWithName + 3436
34  libpython3.6m.dylib                 0x0000000105240f21 fast_function + 657
35  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
36  libpython3.6m.dylib                 0x000000010523c148 _PyEval_EvalFrameDefault + 25080
37  libpython3.6m.dylib                 0x0000000105240e78 fast_function + 488
38  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
39  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
40  libpython3.6m.dylib                 0x000000010524059c _PyEval_EvalCodeWithName + 3436
41  libpython3.6m.dylib                 0x00000001052412bb _PyFunction_FastCallDict + 891
42  libpython3.6m.dylib                 0x000000010516ff93 _PyObject_FastCallDict + 307
43  libpython3.6m.dylib                 0x00000001051700b5 _PyObject_Call_Prepend + 149
44  libpython3.6m.dylib                 0x000000010516fd51 PyObject_Call + 97
45  libpython3.6m.dylib                 0x000000010523c314 _PyEval_EvalFrameDefault + 25540
46  libpython3.6m.dylib                 0x000000010524059c _PyEval_EvalCodeWithName + 3436
47  libpython3.6m.dylib                 0x0000000105235efb PyEval_EvalCodeEx + 107
48  libpython3.6m.dylib                 0x000000010519ad9d function_call + 381
49  libpython3.6m.dylib                 0x000000010516fd51 PyObject_Call + 97
50  libpython3.6m.dylib                 0x000000010523c314 _PyEval_EvalFrameDefault + 25540
51  libpython3.6m.dylib                 0x000000010524059c _PyEval_EvalCodeWithName + 3436
52  libpython3.6m.dylib                 0x0000000105240f21 fast_function + 657
53  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
54  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
55  libpython3.6m.dylib                 0x0000000105240e78 fast_function + 488
56  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
57  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
58  libpython3.6m.dylib                 0x0000000105240e78 fast_function + 488
59  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
60  libpython3.6m.dylib                 0x000000010523c0b8 _PyEval_EvalFrameDefault + 24936
61  libpython3.6m.dylib                 0x000000010524059c _PyEval_EvalCodeWithName + 3436
62  libpython3.6m.dylib                 0x0000000105240f21 fast_function + 657
63  libpython3.6m.dylib                 0x000000010523f731 call_function + 401
*** end of backtrace ***
DAMN ! worker 2 (pid: 26548) died :( trying respawn ...

[xrmx]

It's exploding inside that xpc* stuff, any clue what that is?

[buhrmann]

Very difficult to find info about it, but seem like it's an Apple system library for inter-process communication.

[buhrmann]

Seems related to this on stackoverflow, but neither the option enable-threads nor threads=2 seems to have any effect.

[buhrmann]

To add some further information: the segfault occurs independently of the request library used (both requests and urllib.request), and also when the request is send to another server (e.g. google's). In other words, it seems to be a problem simply with sending any request from within the worker, no matter what kind or where to...

[Ferguzz]

I am experiencing this issue too. i found this related ticket with a temporary workaround https://bugs.python.org/issue30385