Add "download all search results as a tsv"

[lukasjelonek]

The tsv should include all data available in the index.

[lukasjelonek]

Implementation options

naive implementation

Use search to retrieve all entries and compile the tsv on the client (website).

Missing features:

• server does not support pagination past 10.000 entries in a search result. Once this exists, the client could retrieve the whole data and compile the tsv
• Client needs to generate tsv file

Properties:

• Client has control what is part of the tsv

Advantages:

• Besides the pagination above 10.000 entries nothing has to be implemented on the server
• Client can display progress

Disadvantages:

• Requires implementation on the client
• Not sure how much data the client can store before running out of memory
• Will require unneccessarily large data transfers
  • assuming pages with 10.000 elements it will take ~610 Mb uncompressed json datatransfer in the worst case (download all entries)
  • I expect that the tsv will be a lot smaller that the raw json results.

server implementation

The server generates the tsv file in one command and then returns the file as a download

Missing features:

• Server requires a new endpoint that retrieves all search results as tsv
  • paginate through all results for a query
  • generate tsv

Properties:

• Server has control what is part of the tsv

Advantages:

• Only the tsv data is sent to the client
• Can be optimized

Disadvantages:

• Will require more memory on the server
• Request may take a long time and it would require extra code to show some progress on the client

mixed implementation

Add a tsv content-type to the search endpoint and let the client download the tsv.

Missing features:

• convert results to tsv and return the tsv from the search endpoint
• paginate past 10.000 elements
• join and save tsv batches on the client

Properties:

• Server has control what is part of the tsv

Advantages:

• less bandwitdth required than json
• client can decide batch size and display the progress

Disadvantages:

• Feature will be implemented in two places

[lukasjelonek]

I tend to prefer the mixed implementation option.

Ideas:

• Include total/search_after information in first comment line, e.g. ## total: 1003, results:100, search_after: ["xy", 23, "xx"]
• Include header line #id\tgenus\t[...]
• Add option to exclude headers-line to further reduce transfer size
• The client will have to strip the first line and interpret it to get the next page and join all results in a buffer

[lukasjelonek]

The search now does support search_after searches. This enables download of more than 10.000 entries. The next step will be the implementation of a tsv converter.