search

CartoDB / CartoDB-SQL-API

CartoDB SQL API
BSD 3-Clause "New" or "Revised" License
63 stars 64 forks source link

No content-length in response header for shp format #76

Closed strk closed 11 years ago

strk commented 11 years ago

Content-length header must be present in the response.

strk commented 11 years ago

Note that we're currently streaming the SHP format, so we don't know Content-Length in advance, is it really important to have that ? @Ferdev

strk commented 11 years ago

But I do see that we're not really doing the standard chunked transfer, maybe we could move in that direction...

strk commented 11 years ago

I take it back, seeing that Transfer-Encoding: chunked is already sent

Ferdev commented 11 years ago

Why are we using streaming instead of regular file serving? Just asking. It feels weird to download a file without knowing how long will it take. I haven't seen that in any web page for a long time in the Internet.

IMHO, streaming it should be used only when you don't know the size of the resource being downloaded (you know, real-time video transmissions and stuff like that).

I guess we're using streaming for a good reason, but I feel this behaviour "strange".

Of course, this isn't important, just an improvement :)

strk commented 11 years ago

No good reason, other than saving some memory and a disk write. If it's not important I would postpone.

strk commented 11 years ago

We just hit a good reason: out-of-memory conditions on exporting tables with >8M records with CSV export. Doesn't necessarely apply to Shapefile export but the idea is we could have more data to send than fits in memory...

So not sure it'd be an improvement to provide a Content-Length at the risk of being unable to send a result. Unless we want to intentionally limit the result size.

strk commented 11 years ago

So this is a won't fix, we need to keep memory usage low.