Large CSV downloads

[JeltevanBoheemen]

Is your feature request related to a problem? Please describe. Depending on user download limit, very large CSV files can be generated . Creation of these files is chunked, so this poses no problem for the server. However, downloading the files is limited to the time offered by Apache's proxy timeout (by default: 30 seconds). There is no timeout value that would fix this problem, since it is highly dependent on internet connection. For downloads into several hundred MB's, this timeout would be exceedingly large.

Describe the solution you'd like Possible solutions with notes:

• Don't serve large files
  • Take some limit, for example 65.000 (Excel's row limit). Everything above that is beyond the responsibility of I-Analyzer.
• Serve the files directly through Apache
  • No Apache proxy timeouts, because no proxy
  • No possibility to authenticate with Django, would expose (possible) protected data publicly
• Chunk large CSV files based on some row limit, e.g. 50.000. Above that number of rows, generate multiple files. Make frontend download these files separately.
• Use X-SendFile headers
  • Interesting solution. This allows Django to authenticate and decide if a file should be served.
  • Header is added to the response that Apache interprets and uses to directly serve a file, bypassing the proxy.
  • The Apache plugin enabling this hasn't been updated since 2011. Backoffice does not approve.
  • The NGINX equivalent is builtin. Would need to Switch I-Analyzer to NGINX. Backoffice approves.

[jgonggrijp]

I think that in a distant past, a similar question has come up about how much users should be able to download (in the most extreme case). If we can find notes from that past, that might shed some additional light on the discussion (especially on the option "don't serve large files").

"Several hundred MBs" could mean an entire corpus in some cases. It might also depend on the corpus whether we want to facilitate that, since publishers generally are rather protective about who gets to download their corpora.

[lukavdplas]

Re. licences and publishers: see also #1437. I-analyzer is already designed with the idea that access needs to be managed per corpus; I think that should extend to download rights. However, we also serve data where that isn't an issue.