MemoryLeak??

[mmichealjroberts]

Environment details

• OS type and version: macOSCatalina 10.15.7
• Python version: python --version Python 3.8.1
• pip version: pip --version pip 20.2.3 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
• google-api-python-client version: pip show google-api-python-client 1.12.3

There seem to be memory leaks again with this client. Very frustrating. Here are some stack trace examples:

Stack trace

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/src/app/api/surveys/management/commands/survey_results_report.py", line 133, in handle
    result = service.spreadsheets().values().update(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1402, in methodResource
    return Resource(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1289, in __init__
    self._set_service_methods()
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1338, in _set_service_methods
    self._add_basic_methods(self._resourceDesc, self._rootDesc, self._schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1371, in _add_basic_methods
    fixedMethodName, method = createMethod(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1183, in createMethod
    docs.append(schema.prettyPrintSchema(methodDesc["response"]))
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 158, in prettyPrintSchema
    return self._prettyPrintSchema(schema, dent=1)[:-2]
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 145, in _prettyPrintSchema
    return _SchemaToStruct(schema, seen, dent=dent).to_str(self._prettyPrintByName)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 267, in _to_str_impl
    s = self.from_cache(schemaName, seen=self.seen)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 108, in _prettyPrintByName
    self.pretty[name] = _SchemaToStruct(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 258, in _to_str_impl
    self._to_str_impl(pschema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 299, in _to_str_impl
    self.string = "".join(self.value)
MemoryError

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/src/app/api/surveys/management/commands/survey_results_report.py", line 133, in handle
    result = service.spreadsheets().values().update(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1402, in methodResource
    return Resource(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1289, in __init__
    self._set_service_methods()
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1338, in _set_service_methods
    self._add_basic_methods(self._resourceDesc, self._rootDesc, self._schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1371, in _add_basic_methods
    fixedMethodName, method = createMethod(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 1183, in createMethod
    docs.append(schema.prettyPrintSchema(methodDesc["response"]))
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 158, in prettyPrintSchema
    return self._prettyPrintSchema(schema, dent=1)[:-2]
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 145, in _prettyPrintSchema
    return _SchemaToStruct(schema, seen, dent=dent).to_str(self._prettyPrintByName)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 267, in _to_str_impl
    s = self.from_cache(schemaName, seen=self.seen)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 108, in _prettyPrintByName
    self.pretty[name] = _SchemaToStruct(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 258, in _to_str_impl
    self._to_str_impl(pschema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 292, in _to_str_impl
    self._to_str_impl(schema["items"])
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 267, in _to_str_impl
    s = self.from_cache(schemaName, seen=self.seen)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 108, in _prettyPrintByName
    self.pretty[name] = _SchemaToStruct(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 258, in _to_str_impl
    self._to_str_impl(pschema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 292, in _to_str_impl
    self._to_str_impl(schema["items"])
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 267, in _to_str_impl
    s = self.from_cache(schemaName, seen=self.seen)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 108, in _prettyPrintByName
    self.pretty[name] = _SchemaToStruct(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 258, in _to_str_impl
    self._to_str_impl(pschema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 267, in _to_str_impl
    s = self.from_cache(schemaName, seen=self.seen)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 108, in _prettyPrintByName
    self.pretty[name] = _SchemaToStruct(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 315, in to_str
    return self._to_str_impl(self.schema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 258, in _to_str_impl
    self._to_str_impl(pschema)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/schema.py", line 299, in _to_str_impl
    self.string = "".join(self.value)
MemoryError

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
psycopg2.DatabaseError: out of memory for query result

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/src/app/api/surveys/management/commands/survey_results_report.py", line 118, in handle
    for survey_result in survey_results.iterator():
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 346, in _iterator
    yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 74, in __iter__
    for row in compiler.results_iter(results):
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1584, in cursor_iter
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1584, in <lambda>
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
django.db.utils.DatabaseError: out of memory for query result

This is absolutely unacceptable and possibly the first time I have seen this number of Memory errors of this nature from a python package.

[busunkim96]

Hi @michealjroberts,

https://github.com/googleapis/google-api-python-client/issues/535 reports the same issue.

The out of memory seems to come from repeated calls to discovery.build(...). That also has a small benefit for credentials, which will be re-used until they expire.

Copied from https://github.com/googleapis/google-api-python-client/issues/535

service = discovery.build("sheets", "v4")
sheets = service.spreadsheets()
for i in range(0, 50):
    get_responses(sheets)
    sleep(2) # no leak