search

tesserae / apitess

Tesserae API
0 stars 4 forks source link

Enhancement: Endpoint for Search Download #63

Closed jeffkinnison closed 3 years ago

jeffkinnison commented 3 years ago

I propose the following endpoint for search download:

/parallels/<search-id>.<filetype>

This should behave as follows:

GET

Rationale

Even though we are possibly dynamically generating the files, conceptually the formatted version is always found in the same place by referencing the search and a particular file format. This should reduce the complexity of requests by removing the need for encoding URL params. Moreover, we should be able to separate out the download endpoint from the standard /parallels/<search-id>/ by with something like @app.route('/parallels/<path:filename>') and then use os.path.splitext to separate the search id and filetype.

nOkuda commented 3 years ago

My current design is /parallels/<uuid>/tsv/ (since I was intending to implement tab-delimited only for the moment). Additional endpoints could then be added once more file types are supported.

I think we're on the same page about behavior.

nOkuda commented 3 years ago

Implemented endpoint is /parallels/<uuid>/downloads/. Only gzipped tab-delimited is currently supported, though the endpoint seems easily extendible for other formats.

The current implementation creates the gzipped tab-delimited file on the server at the time of the search.

In the future, it could be possible to support other formats by doing one of the following:

  1. generate more files at search time (this is easy to implement but could lead to long search times)
  2. figure out how to more quickly generate files on the fly (working with the database to page results more or less works if we're working with a subset of the results, but leads to long response times when gathering all results for download)
  3. implement an asynchronous approach, where download files are generated only after a user request, similar to the way searches are requested (this would require complicating the database setup with another job type).