speech 0.32.0 causing memory leak

[akmalmzamri]

1. Specify the API google-cloud-speech 0.32.0

2. OS type and version Ubuntu and CentOS

3. Python version 2.7.3

4. Step to reproduce I have a chatbot program where it let user speaks to the computer and computer will response back. The program will call speech to text module from time to time. I notice that after few minutes, the program crash with OSError: [Errno 12] Cannot allocate memory. I ran the program again along side top command and I notice that the memory usage is gradually increasing. After several debugging process, I found out that the problem came from the google-cloud-speech usage. I downgrade the library to the one that I used before (0.29.0) and run the program again. I notice that this time, the memory usage is not that high and it didn't increase over time.

top output when I'm running the program with google-cloud-speech 0.32.0

top output when I'm running the program with google-cloud-speech 0.29.0

Here's the code that I used for this test. memory_leak_test.txt

[tseaver]

@akmalhakimi1991 What version of grpc are you using?

[akmalmzamri]

@tseaver 1.9.1

[theacodes]

@akmalhakimi1991 can you try using grpcio 1.10.0rc2?

[akmalmzamri]

@jonparrott Here's test result after I ran the code for 30 sec:

speech	grpcio	result	max memory %
0.32.1	1.10.0rc2	memory leaked	10++
0.29.0	1.10.0rc2	no leaking	6.1
0.32.1	1.9.1	memory leaked	10++
0.29.0	1.9.1	no leaking	8.0
0.32.1	1.8.3	memory leaked	10++
0.29.0	1.8.3	no leaking	5.3

Here's my pip freeze output:

altgraph==0.15 asn1crypto==0.23.0 aspy.yaml==1.0.0 backports-abc==0.5 backports.ssl-match-hostname==3.5.0.1 cached-property==1.3.1 cachetools==2.0.1 certifi==2018.1.18 cffi==1.11.2 chardet==3.0.4 cheroot==6.0.0 CherryPy==11.2.0 click==6.7 cryptography==2.1.4 dill==0.2.7.1 dis3==0.1.2 enum34==1.1.6 future==0.16.0 futures==3.2.0 google-api-core==1.0.0 google-auth==1.4.1 google-cloud-core==0.27.1 google-cloud-speech==0.32.1 google-cloud-translate==1.3.0 google-gax==0.15.16 googleapis-common-protos==1.5.3 grpcio==1.10.0rc2 gTTS==1.2.2 gTTS-token==1.1.1 identify==1.0.7 idna==2.6 ipaddress==1.0.18 jaraco.classes==1.4.3 lxml==4.1.1 macholib==1.9 monotonic==1.4 more-itertools==4.1.0 nodeenv==1.2.0 pefile==2017.11.5 pluggy==0.6.0 ply==3.8 portend==2.2 pre-commit==1.4.5 protobuf==3.5.2 py==1.5.2 pyasn1==0.4.2 pyasn1-modules==0.2.1 PyAudio==0.2.11 pycparser==2.18 PyInstaller==3.3.1 pyOpenSSL==17.5.0 pyttsx==1.1 pytz==2018.3 PyYAML==3.12 requests==2.18.4 rsa==3.4.2 singledispatch==3.4.0.3 six==1.11.0 SpeechRecognition==3.8.1 tempora==1.10 tenacity==4.7.1 tornado==4.5.2 tox==2.9.1 translate==3.5.0 urllib3==1.22 virtualenv==15.1.0 ws4py==0.4.2

[chemelnucfin]

@akmalhakimi1991 Hello, so it seems like if we fix this issue, #5041 will become a non-issue, is that correct?

Not sure what to make of this yet.
https://gist.github.com/chemelnucfin/f46bf482743452b4723de718519863b5

Let's see if mprof gets me something.

[chemelnucfin]

As a possible workaround, would moving client=speech.SpeechClient() outside of your while True loop work?

[chemelnucfin]

never mind, it still seems to be leaking.

[akmalmzamri]

@chemelnucfin I'm not really sure if this issue is related to #5041. I can't reproduce it so it's hard to say. All I know is, my google search for that error message brings me to GRPCIO

[crwilcox]

@akmalhakimi1991, this should be resolved as we believe the root of the issue was fixed in a recent update to grpcio. If you update grpcio by running pip install grpcio --upgrade your issue should be resolved. This will get version 1.11.0 which has the fix.

[akmalmzamri]

@crwilcox I have yet to updated my grpcio but updating google-cloud-speech to 0.33.0 seems to fix the issue as well