Random Intermittent Segfaults after upgrading from v22.1.0 to v25.0.0

[poojavp95]

Describe the bug: Earlier we were using google-ads=22.1.0 without any issues. Last month we upgraded to google-ads=25.0.0 because of google-ads lib version update and we have started seeing random segfaults happening. We initially used grpc version 1.59.0. Also tried using versions grpc versions 1.59.3, 1.65.5, 1.67.0 etc but all are throwing the same error. Caught a stack trace of one of the segfaults. We cannot downgrade below 1.59.0 as google-ads=25.0.0 requires grpcio >= 1.59.0 Also attaching core dump

core-dump.txt

Thread 0x00007fabd6afd640 (most recent call first):
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 39 in acquire_with_timeout
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 99 in wait
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 220 in get
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run
in run

Thread 0x00007fac111fa640 (most recent call first):
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 39 in acquire_with_timeout
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 99 in wait
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/_threading.py", line 220 in get
  File "/segment-source/venv/lib/python3.10/site-packages/gevent/threadpool.py", line 195 in run

Current thread 0x00007fac36e19000 (most recent call first):
  File "/segment-source/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1142 in _blocking
  File "/segment-source/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1158 in __call__
  File "/segment-source/venv/lib/python3.10/site-packages/segment_source/client.py", line 44 in set
  File "/segment-source/venv/lib/python3.10/site-packages/integration/resources/base.py", line 129 in set
  File "/segment-source/venv/lib/python3.10/site-packages/integration/resources/ad_performance_reports.py", line 47 in download

Extension modules: greenlet._greenlet, zope.interface._zope_interface_coptimizations, gevent.libev.corecext, gevent._gevent_c_greenlet_primitives, gevent._gevent_c_hub_local, gevent._gevent_c_waiter, gevent._gevent_c_hub_primitives, gevent._gevent_c_ident, gevent._gevent_cgreenlet, gevent._gevent_c_abstract_linkable, gevent._gevent_c_semaphore, gevent._gevent_clocal, gevent._gevent_cevent, gevent._gevent_cqueue, grpc._cython.cygrpc, gevent._gevent_c_imap, google._upb._message, yaml._yaml (total: 18)
{"level":"ERROR","time":"2024-10-23T07:12:13.073198746Z","info":{},"data":{"error":"signal: segmentation fault (core dumped)","job_name":"","program":"source-runner","source":"(*Source).runApplication: /go/src/github.com/segmentio/source-runner/source.go:678","version":"c9243b8"},"message":"error occurred during this sync"}
{"level":"DEBUG","time":"2024-10-23T07:12:13.073287259Z","info":{},"data":{"exit_code":1,"job_name":"","program

Steps to Reproduce: This error is happening after our python job runs for an hour or more and that too intermittently. All the jobs running less than an hour run successfully.

Expected behavior: No segfault

Client library version and API version: Client library version: 25.0.0 Google Ads API version: v17

argcomplete==1.12.3 attrs==22.2.0 cachetools==5.0.0 certifi==2021.10.8 charset-normalizer==2.0.12 colorlog==6.6.0 coverage==7.1.0 decorator==5.1.1 distlib==0.3.4 filelock==3.6.0 gevent==22.10.2 gevent-semaphore-decorator==0.2.0 google-ads==25.0.0 google-api-core==2.13.0 google-auth==2.16.0 google-auth-oauthlib==0.8.0 googleapis-common-protos==1.56.4 greenlet==2.0.0 grpcio==1.59.0 grpcio-status==1.59.0 idna==3.3 importlib-metadata==4.11.3 iniconfig==2.0.0 nox==2022.1.7 oauthlib==3.2.0 packaging==21.3 platformdirs==2.5.1 pluggy==1.0.0 proto-plus==1.22.3 protobuf==4.25.0 py==1.11.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pydash==5.1.0 pyparsing==3.0.7 pytest==7.0.1 pytest-cov==3.0.0 python-dateutil==2.8.2 PyYAML==6.0.1 requests==2.27.1 requests-oauthlib==1.3.1 retry==0.9.2 rsa==4.8 six==1.16.0 structlog==21.2.0 tomli==2.0.1 typing_extensions==4.1.1 urllib3==1.26.8 virtualenv==20.13.3 zipp==3.7.0 zope.event==4.6 zope.interface==5.5.2 setuptools==58.0.4

Request/Response Logs:

Anything else we should know about your project / environment:

[BenRKarl]

@poojavp95 thanks for the details - just curious if you get different results when using newer versions of grpcio, can you try manually installing a newer version?

[BenRKarl]

Closing this issue in favor of the related thread here: https://github.com/grpc/grpc/issues/37994