Open johnthagen opened 1 year ago
Interesting that if I read the file into memory and use a vanilla Django HttpResponse
directly from a DRF ViewSet
, it transfers extremely quickly.
class MyModelViewSet(RetrieveModelMixin, GenericViewSet):
...
def retrieve(self, request: Request, *args: Any, **kwargs: str) -> HttpResponse:
my_model = self.get_object()
export_data = Path(my_model.model.path).read_bytes()
return HttpResponse(
export_data,
content_type= "application/octet-stream",
headers={
"Content-Disposition": "attachment; "
'filename="out.bin"',
},
)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 118M 100 118M 0 0 162M 0 --:--:-- --:--:-- --:--:-- 161M
I think that is roughly 1.3Gbps. I'm seeing anywhere around a 40x speed improvement.
I suspect that the underlying issue is inefficiency in how Django/Python/WSGI stream bytes by looping over the input in the Python interpreter:
The way I solved this was to simply use pure DRF views to serve files rather than django-downloadview
, since I haven't been able to get NGINX acceleration working:
In case this helps others, here is an implementation:
from pathlib import Path
from typing import Any
from django.db.models import Model
from django.forms import FileField
from django.http import HttpResponse
from drf_spectacular.utils import extend_schema, extend_schema_view
from rest_framework.filters import BaseFilterBackend
from rest_framework.mixins import RetrieveModelMixin
from rest_framework.renderers import BaseRenderer
from rest_framework.request import Request
from rest_framework.viewsets import GenericViewSet
class BinaryRenderer(BaseRenderer):
"""A renderer that supports binary file downloads."""
media_type = "application/octet-stream"
format = "bin"
def render(
self,
data: bytes,
accepted_media_type: str | None = None,
renderer_context: dict[str, Any] | None = None,
) -> bytes:
return data
@extend_schema_view(retrieve=extend_schema(responses=bytes))
class FileDownloadViewSet(RetrieveModelMixin, GenericViewSet):
"""A generic ViewSet that supports serving downloaded files.
Rather than streaming the file, the entire file is read into memory and sent out, which can
improve the download speed of files at the cost of maximum memory.
Fill in the `file_field` and `filename_field` attributes when deriving from this class to set
which file should be downloaded from the Model.
"""
filter_backends: list[BaseFilterBackend] = []
renderer_classes = [BinaryRenderer]
file_field: str = NotImplemented
filename_field: str = NotImplemented
def retrieve(self, request: Request, *args: Any, **kwargs: str) -> HttpResponse:
model: Model = self.get_object()
file: FileField = getattr(model, self.file_field)
file_data = Path(file.path).read_bytes() # type: ignore[attr-defined]
file_name: str = getattr(model, self.filename_field)
return HttpResponse(
file_data,
content_type=BinaryRenderer.media_type,
headers={"Content-Disposition": f'attachment; filename="{file_name}"'},
)
And an example usage:
class MyModelDownloadViewSet(FileDownloadViewSet):
queryset = MyModel.objects.all()
permission_classes = [DjangoObjectPermissions]
file_field = "file"
filename_field = "filename"
I have an
ObjectDownloadView
that is serving very large files (200MB - 2GB). I've observed that the download speed is very slow, even when pulling downloads over localhost where no actual network is involved at all. I must authenticate the file downloads (not shown the example below), which is why I must usedjango-downloadview
rather than serving them statically.I cannot use NGINX acceleration due to:
177
My endpoint looks something like:
The
Model
:URLs:
I've tested downloading the file using a variety of clients over localhost (also running locally on mac and also within a Linux Docker container) and they all show a similar result:
httpx
This corresponds to about 40Mbps, which seemed very slow for a localhost pull. I also see the
python
executable running at about 100% CPU, as if it's CPU rather than I/O bound?Is there something about how
django-downloadview
streams or chunks the file that contributes to why this is so slow?Are there any configuration settings to speed up serving files natively from
django-downloadview
?