search

mjumbewu / django-rest-framework-csv

CSV Tools for Django REST Framework
BSD 2-Clause "Simplified" License
364 stars 89 forks source link

Is it possible to use StreamingHttpResponse? #16

Closed buddylindsey closed 8 years ago

buddylindsey commented 10 years ago

We are currently using django-rest-framework-csv for csv export, but we have found we have some large datasets that eventually lead to a timeout between nginx and our app. Is it possible to use StreamingHttpResponse for returning a csv file using this library?

mjumbewu commented 10 years ago

This is something I've wanted to do for a while, as I've run into issues with large datasets as well, and CSV lends itself so well to a streaming response.

It's not currently possible with drf-csv because the renderer builds up the entire table before returning. This helps it play nicely with DRF's default Response objects. For a streaming response, I could imagine a similar method to the existing render (maybe streaming_render) that just returns a generator instead of a string. All the other pieces of the equation should already exist (instead of calling tablize, this streaming method would likely duplicate what gets done in tablize; some refactoring would be in order).

The other aspect that might contribute to timeouts is the logic for inferring the table's columns. I can see offhand a number of places where this process could be optimized (e.g. using comprehensions and iterators where appropriate, and not wasting time "flattening" stuff that not dict or list).

ivancrneto commented 10 years ago

Please see #20

frewsxcv commented 8 years ago

https://github.com/mjumbewu/django-rest-framework-csv/blob/ee9a05e103c5d1c08216ef10449a25163ff1d1b1/rest_framework_csv/renderers.py#L183

There is now a CSVStreamingRenderer. This can probably be closed?