Random segfaults in grpc python after upgrading to 1.59.x and above

What version of gRPC and what language are you using?

grpicio=1.59.0+ , Python 3.10

What operating system (Linux, Windows,...) and version?

Linux ( Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1068-aws x86_64))

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.10 (Same error with python 3.9.6 as well)

What did you do?

Earlier we were using grpcio=1.48.0 without any issues. Last month we upgraded to grpcio=1.59.0 because of google-ads lib version update and we have started seeing random segfaults happening. Tried using versions 1.59.3, 1.65.5, 1.67.0 etc but all are throwing the same error. Caught a stack trace of one of the segfaults. We cannot downgrade below 1.59.0 as google-ads requires grpcio >= 1.59.0

Thread 0x00007fabd6afd640 (most recent call first):
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 39 in acquire_with_timeout
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 99 in wait
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 220 in get
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run
in run

Thread 0x00007fac111fa640 (most recent call first):
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 39 in acquire_with_timeout
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 99 in wait
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 220 in get
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run

Current thread 0x00007fac36e19000 (most recent call first):
  File "/segment-source/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1142 in _blocking
  File "/segment-source/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1158 in __call__
  File "/segment-source/venv/lib/python3.10/site-packages/segment_source/client.py", line 44 in set
  File "/segment-source/venv/lib/python3.10/site-packages/integration/resources/base.py", line 129 in set
  File "/segment-source/venv/lib/python3.10/site-packages/integration/resources/ad_performance_reports.py", line 47 in download

Extension modules: greenlet._greenlet, zope.interface._zope_interface_coptimizations, gevent.libev.corecext, gevent._gevent_c_greenlet_primitives, gevent._gevent_c_hub_local, gevent._gevent_c_waiter, gevent._gevent_c_hub_primitives, gevent._gevent_c_ident, gevent._gevent_cgreenlet, gevent._gevent_c_abstract_linkable, gevent._gevent_c_semaphore, gevent._gevent_clocal, gevent._gevent_cevent, gevent._gevent_cqueue, grpc._cython.cygrpc, gevent._gevent_c_imap, google._upb._message, yaml._yaml (total: 18)
{"level":"ERROR","time":"2024-10-23T07:12:13.073198746Z","info":{},"data":{"error":"signal: segmentation fault (core dumped)","job_name":"","program":"source-runner","source":"(*Source).runApplication: /go/src/github.com/segmentio/source-runner/source.go:678","version":"c9243b8"},"message":"error occurred during this sync"}
{"level":"DEBUG","time":"2024-10-23T07:12:13.073287259Z","info":{},"data":{"exit_code":1,"job_name":"","program":"source-runner","version":"c9243b8"},"message":"Stopping...source exit code 1"}

What did you expect to see?

No segfault

What did you see instead?

Segmentation fault

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

grpc / grpc