MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

job exports download memory limits #207

Closed ghukill closed 6 years ago

ghukill commented 6 years ago

Appears there is potential for memory limit on download, received following error on downloading 350k+ export:

Environment:

Request Method: GET
Request URL: http://foo.com/combine/document_download?filepath=/tmp/af1c6f76-5c38-4628-a215-9c16e5d7058c/j_148_documents.zip&name=j_148_documents.zip&content_type=application/zip

Django Version: 1.11.4
Python Version: 3.5.5
Installed Applications:
['core',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django_extensions',
 'background_task']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/usr/local/Miniconda3-4.3.30-Linux-x86_64/envs/combine/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/local/Miniconda3-4.3.30-Linux-x86_64/envs/combine/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/Miniconda3-4.3.30-Linux-x86_64/envs/combine/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/opt/combine/core/views.py" in document_download
  1638.         response = HttpResponse(fhand, content_type=content_type)

File "/usr/local/Miniconda3-4.3.30-Linux-x86_64/envs/combine/lib/python3.5/site-packages/django/http/response.py" in __init__
  303.         self.content = content

File "/usr/local/Miniconda3-4.3.30-Linux-x86_64/envs/combine/lib/python3.5/site-packages/django/http/response.py" in content
  329.             content = b''.join(self.make_bytes(chunk) for chunk in value)

Exception Type: MemoryError at /combine/document_download
Exception Value: 
ghukill commented 6 years ago

Confirmed with attempting to download zip file of 1.3g. Investigating...

ghukill commented 6 years ago

Fixed by using Django's built-in FileResponse. Closing.