greenlet.switch triggers address sanitizer

tbodt commented 7 years ago

Here's the output:

==17897==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7fff5a996440 at pc 0x00010580e9d6 bp 0x7fff5a996340 sp 0x7fff5a995ae8
READ of size 17424 at 0x7fff5a996440 thread T0
    #0 0x10580e9d5  (libclang_rt.asan_osx_dynamic.dylib+0x479d5)
    #1 0x1101b1e05 in slp_save_state (greenlet.so+0x2e05)
    #2 0x1101b1d1c in slp_switch (greenlet.so+0x2d1c)
    #3 0x1101b1431 in g_switch (greenlet.so+0x2431)
    #4 0x1101b2005 in green_switch (greenlet.so+0x3005)
    #5 0x105445f03 in PyEval_EvalFrameEx ceval.c:4352
    #6 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #7 0x1052dcce3 in function_call funcobject.c:523
    #8 0x10526bf28 in PyObject_Call abstract.c:2547
    #9 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    <my code. snip>
    #23 0x105445f03 in PyEval_EvalFrameEx ceval.c:4352
    #24 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #25 0x1052dcce3 in function_call funcobject.c:523
    #26 0x10526bf28 in PyObject_Call abstract.c:2547
    #27 0x106db4649 in partial_call _functoolsmodule.c:197
    #28 0x10526bf28 in PyObject_Call abstract.c:2547
    #29 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    #30 0x1101b1b29 in g_initialstub (greenlet.so+0x2b29)
    #31 0x1101b13ac in g_switch (greenlet.so+0x23ac)
    #32 0x1101b2005 in green_switch (greenlet.so+0x3005)
    #33 0x105445f03 in PyEval_EvalFrameEx ceval.c:4352
    #34 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #35 0x1052dcce3 in function_call funcobject.c:523
    #36 0x10526bf28 in PyObject_Call abstract.c:2547
    #37 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    <more of my code. snip>
    #51 0x105445f03 in PyEval_EvalFrameEx ceval.c:4352
    #52 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #53 0x1052dcce3 in function_call funcobject.c:523
    #54 0x10526bf28 in PyObject_Call abstract.c:2547
    #55 0x106db4649 in partial_call _functoolsmodule.c:197
    #56 0x10526bf28 in PyObject_Call abstract.c:2547
    #57 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    #58 0x1101b1b29 in g_initialstub (greenlet.so+0x2b29)
    #59 0x1101b13ac in g_switch (greenlet.so+0x23ac)
    #60 0x1101b2005 in green_switch (greenlet.so+0x3005)
    #61 0x105445f03 in PyEval_EvalFrameEx ceval.c:4352
    #62 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #63 0x1052dcce3 in function_call funcobject.c:523
    #64 0x10526bf28 in PyObject_Call abstract.c:2547
    #65 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #66 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #67 0x1052dcce3 in function_call funcobject.c:523
    #68 0x10526bf28 in PyObject_Call abstract.c:2547
    #69 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #70 0x10545c18e in fast_function ceval.c:4437
    #71 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #72 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #73 0x1052dcce3 in function_call funcobject.c:523
    #74 0x10526bf28 in PyObject_Call abstract.c:2547
    #75 0x105297b4c in instancemethod_call classobject.c:2602
    #76 0x10526bf28 in PyObject_Call abstract.c:2547
    #77 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    #78 0x10529183f in PyInstance_New classobject.c:581
    #79 0x10526bf28 in PyObject_Call abstract.c:2547
    #80 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #81 0x10545c18e in fast_function ceval.c:4437
    #82 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #83 0x10545c18e in fast_function ceval.c:4437
    #84 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #85 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #86 0x10545bf6a in fast_function ceval.c:4447
    #87 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #88 0x10545c18e in fast_function ceval.c:4437
    #89 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #90 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #91 0x1052dcce3 in function_call funcobject.c:523
    #92 0x10526bf28 in PyObject_Call abstract.c:2547
    #93 0x105297b4c in instancemethod_call classobject.c:2602
    #94 0x10526bf28 in PyObject_Call abstract.c:2547
    #95 0x105381936 in slot_tp_call typeobject.c:5546
    #96 0x10526bf28 in PyObject_Call abstract.c:2547
    #97 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #98 0x10545c18e in fast_function ceval.c:4437
    #99 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #100 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #101 0x1052dcce3 in function_call funcobject.c:523
    #102 0x10526bf28 in PyObject_Call abstract.c:2547
    #103 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #104 0x10545c18e in fast_function ceval.c:4437
    #105 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #106 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #107 0x1052dcce3 in function_call funcobject.c:523
    #108 0x10526bf28 in PyObject_Call abstract.c:2547
    #109 0x105297b4c in instancemethod_call classobject.c:2602
    #110 0x10526bf28 in PyObject_Call abstract.c:2547
    #111 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    #112 0x10529183f in PyInstance_New classobject.c:581
    #113 0x10526bf28 in PyObject_Call abstract.c:2547
    #114 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #115 0x10545c18e in fast_function ceval.c:4437
    #116 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #117 0x10545c18e in fast_function ceval.c:4437
    #118 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #119 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #120 0x10545bf6a in fast_function ceval.c:4447
    #121 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #122 0x10545c18e in fast_function ceval.c:4437
    #123 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #124 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #125 0x1052dcce3 in function_call funcobject.c:523
    #126 0x10526bf28 in PyObject_Call abstract.c:2547
    #127 0x105297b4c in instancemethod_call classobject.c:2602
    #128 0x10526bf28 in PyObject_Call abstract.c:2547
    #129 0x105381936 in slot_tp_call typeobject.c:5546
    #130 0x10526bf28 in PyObject_Call abstract.c:2547
    #131 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #132 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #133 0x10545bf6a in fast_function ceval.c:4447
    #134 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #135 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #136 0x1052dcce3 in function_call funcobject.c:523
    #137 0x10526bf28 in PyObject_Call abstract.c:2547
    #138 0x105297b4c in instancemethod_call classobject.c:2602
    #139 0x10526bf28 in PyObject_Call abstract.c:2547
    #140 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    #141 0x10529183f in PyInstance_New classobject.c:581
    #142 0x10526bf28 in PyObject_Call abstract.c:2547
    #143 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #144 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #145 0x1052dcce3 in function_call funcobject.c:523
    #146 0x10526bf28 in PyObject_Call abstract.c:2547
    #147 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #148 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #149 0x10545bf6a in fast_function ceval.c:4447
    #150 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #151 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #152 0x10545bf6a in fast_function ceval.c:4447
    #153 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #154 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #155 0x1052dcce3 in function_call funcobject.c:523
    #156 0x10526bf28 in PyObject_Call abstract.c:2547
    #157 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #158 0x10545c18e in fast_function ceval.c:4437
    #159 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #160 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #161 0x1052dcce3 in function_call funcobject.c:523
    #162 0x10526bf28 in PyObject_Call abstract.c:2547
    #163 0x105297b4c in instancemethod_call classobject.c:2602
    #164 0x10526bf28 in PyObject_Call abstract.c:2547
    #165 0x10545aeb3 in PyEval_CallObjectWithKeywords ceval.c:4221
    #166 0x10529183f in PyInstance_New classobject.c:581
    #167 0x10526bf28 in PyObject_Call abstract.c:2547
    #168 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #169 0x10545c18e in fast_function ceval.c:4437
    #170 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #171 0x10545c18e in fast_function ceval.c:4437
    #172 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #173 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #174 0x10545bf6a in fast_function ceval.c:4447
    #175 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #176 0x10545c18e in fast_function ceval.c:4437
    #177 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #178 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #179 0x1052dcce3 in function_call funcobject.c:523
    #180 0x10526bf28 in PyObject_Call abstract.c:2547
    #181 0x105297b4c in instancemethod_call classobject.c:2602
    #182 0x10526bf28 in PyObject_Call abstract.c:2547
    #183 0x105381936 in slot_tp_call typeobject.c:5546
    #184 0x10526bf28 in PyObject_Call abstract.c:2547
    #185 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #186 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #187 0x1052dcce3 in function_call funcobject.c:523
    #188 0x10526bf28 in PyObject_Call abstract.c:2547
    #189 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #190 0x10545c18e in fast_function ceval.c:4437
    #191 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #192 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #193 0x10545bf6a in fast_function ceval.c:4447
    #194 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #195 0x10545c18e in fast_function ceval.c:4437
    #196 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #197 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #198 0x1052dcce3 in function_call funcobject.c:523
    #199 0x10526bf28 in PyObject_Call abstract.c:2547
    #200 0x105297b4c in instancemethod_call classobject.c:2602
    #201 0x10526bf28 in PyObject_Call abstract.c:2547
    #202 0x105381936 in slot_tp_call typeobject.c:5546
    #203 0x10526bf28 in PyObject_Call abstract.c:2547
    #204 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #205 0x10545c18e in fast_function ceval.c:4437
    #206 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #207 0x10545c18e in fast_function ceval.c:4437
    #208 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #209 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #210 0x1052dcce3 in function_call funcobject.c:523
    #211 0x10526bf28 in PyObject_Call abstract.c:2547
    #212 0x10544870e in PyEval_EvalFrameEx ceval.c:4666
    #213 0x10545c18e in fast_function ceval.c:4437
    #214 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #215 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #216 0x10545bf6a in fast_function ceval.c:4447
    #217 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #218 0x10545c18e in fast_function ceval.c:4437
    #219 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #220 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #221 0x1052dcce3 in function_call funcobject.c:523
    #222 0x10526bf28 in PyObject_Call abstract.c:2547
    #223 0x105297b4c in instancemethod_call classobject.c:2602
    #224 0x10526bf28 in PyObject_Call abstract.c:2547
    #225 0x105381936 in slot_tp_call typeobject.c:5546
    #226 0x10526bf28 in PyObject_Call abstract.c:2547
    #227 0x105445cac in PyEval_EvalFrameEx ceval.c:4569
    #228 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #229 0x10545bf6a in fast_function ceval.c:4447
    #230 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #231 0x10545c18e in fast_function ceval.c:4437
    #232 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #233 0x10545c18e in fast_function ceval.c:4437
    #234 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #235 0x10545c18e in fast_function ceval.c:4437
    #236 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #237 0x10545c18e in fast_function ceval.c:4437
    #238 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #239 0x10545c18e in fast_function ceval.c:4437
    #240 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #241 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #242 0x10545bf6a in fast_function ceval.c:4447
    #243 0x10544596b in PyEval_EvalFrameEx ceval.c:4372
    #244 0x10543d757 in PyEval_EvalCodeEx ceval.c:3584
    #245 0x10543c271 in PyEval_EvalCode ceval.c:669
    #246 0x1054cb84c in PyRun_FileExFlags pythonrun.c:1376
    #247 0x1054caa58 in PyRun_SimpleFileExFlags pythonrun.c:948
    #248 0x105508d71 in Py_Main main.c:640
    #249 0x7fffc8cb6254 in start (libdyld.dylib+0x5254)

Address 0x7fff5a996440 is located in stack of thread T0 at offset 0 in frame
    #0 0x10543e1cf in PyEval_EvalFrameEx ceval.c:690

  This frame has 18 object(s):
    [32, 36) 'cf.i' <== Memory access at offset 0 partially underflows this variable
    [48, 56) 'str.i' <== Memory access at offset 0 partially underflows this variable
    [80, 84) 'cf113.i' <== Memory access at offset 0 partially underflows this variable
    [96, 104) 'type.i' <== Memory access at offset 0 partially underflows this variable
    [128, 136) 'value.i' <== Memory access at offset 0 partially underflows this variable
    [160, 168) 'traceback.i' <== Memory access at offset 0 partially underflows this variable
    [192, 200) 'ptype.i' <== Memory access at offset 0 partially underflows this variable
    [224, 232) 'pvalue.i' <== Memory access at offset 0 partially underflows this variable
    [256, 264) 'ptraceback.i' <== Memory access at offset 0 partially underflows this variable
    [288, 296) 'type.addr.i' <== Memory access at offset 0 partially underflows this variable
    [320, 328) 'value.addr.i' <== Memory access at offset 0 partially underflows this variable
    [352, 360) 'tb.addr.i' <== Memory access at offset 0 partially underflows this variable
    [384, 392) 'bounds.i' <== Memory access at offset 0 partially underflows this variable
    [416, 424) 'sp' <== Memory access at offset 0 partially underflows this variable
    [448, 456) 'sp5779' <== Memory access at offset 0 partially underflows this variable
    [480, 488) 'exc' <== Memory access at offset 0 partially underflows this variable
    [512, 520) 'val' <== Memory access at offset 0 partially underflows this variable
    [544, 552) 'tb' <== Memory access at offset 0 partially underflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow (libclang_rt.asan_osx_dynamic.dylib+0x479d5)
Shadow bytes around the buggy address:
  0x1fffeb532c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffeb532c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffeb532c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffeb532c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffeb532c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1fffeb532c80: 00 00 00 00 00 00 00 00[f1]f1 f1 f1 04 f2 00 f2
  0x1fffeb532c90: f2 f2 04 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
  0x1fffeb532ca0: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
  0x1fffeb532cb0: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
  0x1fffeb532cc0: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f3 f3 f3
  0x1fffeb532cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==17897==ABORTING

Probably the most important line from that:

HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext

snaury commented 7 years ago

I notice you have v8 on your stack. As far as I know greenlet is fundamentally incompatible with v8 (or any C++ code that shares pointers to objects on the stack globally). The reason for this is that when greenlets switch the stack is overwritten back and forth and objects become temporarily invalid until you switch back to that greenlet, thus any access to shared objects becomes a hazard.

In this instance there might be another issue in that greenlet truly does overwrite the stack and it might not look much different from a buffer overflow to a sanitizer. You probably need to patch address sanitizer to teach it how to deal with stack switching in greenlet.

tbodt commented 7 years ago

I know that greenlet is incompatible with V8.

This is not just about V8 though, it also happens when V8 is not involved.

The asan runtime includes functions that stack switchers can call to shut up these messages, you could probably use those.

https://github.com/llvm-mirror/compiler-rt/blob/0b95585616bd28fc0b738289bcc5f7887d7c304e/include/sanitizer/common_interface_defs.h#L164-L184

tbodt commented 7 years ago

Regarding V8 compatibility, is it theoretically possible to reimplement greenlet on top of libcoro? node-fibers uses that, and it works.

snaury commented 7 years ago

My big concerns are stack sizes, switch performance and platform compatibility. For me a big appeal of greenlet was that it doesn't use dedicated stacks, you only pay for the delta of used stack space which makes it possible to create millions of greenlets and they cost very little. Next, swapcontext is very slow and needs a lot of space for all that state, but dedicated stacks mean you need to worry about their sizes. If the size is too small (Coro on perl uses something like 64-128kb, which is a lot btw) you risk running out and crashing, if the size is too big you'd have to use mmap for efficiency and then you run other risks, like running out of the maximum number of mmaps per process.

Personally I was thinking about rewriting greenlet in cython and using boost::context for quite a while, however such a library wouldn't be called greenlet (maybe greenlet-context or whatever, since the number of supported platforms would be much smaller than what greenlet currently supports and what people surprisingly actually use), and I'm not interested in it that much. If you have spare time you may try going that route, however I wouldn't accept such a solution into the mainline greenlet, as it would have entirely different set of drawbacks than what people currently expect.

snaury commented 7 years ago

The asan runtime includes functions that stack switchers can call to shut up these messages, you could probably use those.

I'm not sure those apply, since greenlet doesn't actually switch stacks, it reuses system stack by overwriting it with saved deltas. Currently it doesn't even know where the bottom or what the size of the stack is, so it doesn't look like those callbacks apply to what greenlet does.

tbodt commented 7 years ago

libcoro has a variety of different implementations:

Using assembler to switch esp, works on unix+windows and both 32 and 64 bit intel chips
Using a strange combination of sigaltstack and setjmp that works on any unix where those calls are available
Using ucontext on unix if supported
Using fibers on windows
If all else fails, using pthread

You can choose at compile time. So I don't think platform support is a problem.

About performance:

The assembler-based version uses the minimal amount of instructions to swap out esp and restore registers.
The sigaltstack/setjmp version is pretty slow to create coroutines, because it has to send a signal to itself to trigger sigaltstack creation, but longjmp probably isn't doing a lot more than the custom assembler is.
The rest are pretty slow, but the first two look to me like they would cover most platforms in use. Correct me if I'm wrong.

Stack space is definitely still an issue. FWIW libcoro includes stack allocation code that uses mmap if it's supported and malloc otherwise.

I'd like to make a libcoro-based version of greenlet, but the problem is getting libraries such as gevent to use it. My choices are to either make the fork appear in sys.modules under the name greenlet (which I really don't want to do), or to get something merged into greenlet. If I could pull off something that lets you choose if you want to use libcoro on a per-coroutine basis, could that be merged, or would it be too disgusting?

tbodt commented 7 years ago

And you're right, the asan functions don't really apply to greenlet. They would apply to libcoro though.

snaury commented 7 years ago

It's not about disgust, if you make a fork I would probably gladly start using it myself, and if you convince gevent and eventlet to use it, then even better! :) It would work better, would have less opportunities for crashing and would probably be a lot faster too.

But as a maintainer I feel my primary responsibility is not breaking code for people that already use greenlet for whatever scenario. And scenarios for using greenlet are different for different people too, just 4-5 years ago I had something like 100k greenlets sloshing around in a single process and I have serious doubts any mmap based solution would have worked for me back then. Now that Go is more mature I would have preferred to use Go, and wouldn't want to do anything like that in Python ever again, so I personally wouldn't be affected. However I'm sure there's someone out there who's doing something crazy like that right now and mmap based solution would stop them from succeeding.

Making a patch that makes the new mode optional (behind something like greenlet.enable_awesome_mode()) would probably work, but it would add a lot of complexity to an already complex code, and for what? People won't even know they need to use it, and I have no way of checking all 18+ platforms (and many more variants) for breakage. New library and a clean break is a lot better, at least that's how I would have done it if I was still interested enough. There's way too much complexity in greenlet right now that has to deal with unexpected gc and unexpected switching, with libcoro you'd have to deal with a lot less surprises like that, since stack cannot get invalidated right under you. It would be a lot faster too (no malloc and copy on every switch), but I'm not the one you need to convince.

I think if you make a new library and it's a lot faster (and trust me it will be, 6500ns on benchmarks/chain.py shouldn't be hard to beat), then people would recognize safety and a different set of tradeoffs are worth it. If you make using it as easy as from something import greenlet, then porting and trying it out should be easy. You shouldn't want to integrate it into the current mess of greenlet.

tbodt commented 7 years ago

Sounds like a fun weekend project. 😄

tbodt commented 7 years ago

Make greenlets great again!

navytux commented 5 years ago

Sounds like a fun weekend project. Make greenlets great again!

@tbold, just curious, did anything happened on your side on this topic?

Thanks beforehand for feedback, Kirill

tbodt commented 5 years ago

I made this: http://github.com/tbodt/greenstack. Try not to use it in production.

navytux commented 5 years ago

@tbodt, thanks a lot for feedback. It would be intresting to try to switch to something like greenstack in Pygolang context eventually.

python-greenlet / greenlet

greenlet.switch triggers address sanitizer #113