Add a transcoding queue for large transcoding jobs

[danielballan]

For datasets up to some byte size, GET /array/full/some_array simply returns the data directly. For datasets above that size, we reach one or two limitations:

• The dataset takes so long to transcode that HTTP proxies (e.g. nginx) and/or clients may time out.
• The dataset consumes too much of server RAM.

Therefore, we must transcode it to some persistence storage (e.g. POSIX filesystem, blob storage) and, later, spool it from there. It can be retained for some time to make subsequent requests go faster, as long as space allows.

The client-side experience will be much like when you download a directory of files from Google Drive. Google packs up the files into a zip archive and then, when that's complete, gives you a download link. The proposed flow is:

GET /array/full/very_large_array -> 202 Accepted Location: /queue/{job_id} GET /queue/{job_id} -> 200 OK (body contains stats like how long it has been running) Later... GET /queue/{job_id} -> 303 See Other Location: /download/{job_id} GET /download/{job_id} -> 200 OK (body is large data) Later... GET /download/{job_id} -> 410 Gone (artifact has been garbage collected)

informed by https://farazdagi.com/2014/rest-and-long-running-jobs/ which has disappeared but is still available at https://web.archive.org/web/20200511220257/https://farazdagi.com/2014/rest-and-long-running-jobs/.

We made a very rough prototype of this flow in https://github.com/bluesky/suitcase-server/blob/master/suitcase_server/handlers.py. It has a lot of problems and never got as far as being used but it may be a useful reference.

Requirements:

• [ ] A simple "scaled down" implementation that uses Starlette BackgroundTasks
• [ ] A scalable implementation that uses a shared task queue --- potentially pluggable to Celery, Luigi, etc. in the future --- and a worker pool

This is a prerequisite for #43 because trees are likely to be large --- i.e. "Download everything below this node as an HDF5 file".

[dylanmcreynolds]

I think this is great for a lot of reasons, including the ability to grab large data sets in a batch style mode to produce training sets for AI models...something that @kleinhenz has talked about.

[dylanmcreynolds]

For notifying the client that the package is ready, what about supporting something that can generate events and be consumed by https://developer.mozilla.org/en-US/docs/Web/API/EventSource?

[danielballan]

Oh, cool, I wasn't aware of that feature. At first glance, it looks like this would require separate client capability. For example, there are Python libraries for it but httpx and requests do not encompass it. I guess it also isn't something you could use from curl.

How would you feel about starting with the lo-fi 202 / 303 implementation and then adding the EventSource option in a follow-up?

[dylanmcreynolds]

Sure. The EventSource mechanism seems like a handy way to get one way notification from the server back to clients. The main difference seems to me that 202/303 requires the client poll regularly...EventSouce takes advantage of a long-lasting connection. In both cases, the client is going to have to do something in the background, which I suspect is the real new thing.

Again, the spec is new to me, but I suspect that we're going to want to have done EventSource if/when someone gets serious about developing browser-based apps that talk to Tiled.

[danielballan]

I agree. And I could be persuaded to flip the order. I guess I'm just advocating that we lay track for doing both, because I think will be eventually want both.

[dylanmcreynolds]

I agree with that. And on both the client and server, I'll bet that the two implementations share a lot of track.

[danielballan]

Circling back to think about this almost a year later…

• The idea of using BackgroundTask for this was misguided. This is fundamentally decoupled for the HTTP and we should build it with a shared job queue from the start.
• Permissions are tricky, as users may be able to see different parts of the same node. To start, a given job should be only accessible to the user who initiated it.
• We need a new (SQL?) database to manage this, acting as a work queue and policing things like “same user cannot queue more than exports of the same node to a given format”. The interface could be pluggable to enable integration with external workflow systems like Celery or Prefect but the “work” is so constrained in this case that I question the value of adding those systems. A tightly integrated database might be perfect.
• Time-based expiry seems a pragmatic way to start.
• A long-running work process should consume from the queue. Separately, a single shot transcoding CLI may be a useful thing to have for working inside other workflow managers.

[danielballan]

Also, at some point we discussed reusing artifacts across repeated requests but this is also complicated by permission—the way different users have different views into a node. We would also have to handle the possibility of mutation/modification.

Each asset should be bound to a specific job to start.

[danielballan]

Summarizing discussion with @dylanmcreynolds

• We can use Redis to manage both the queue of jobs and the "database" of outputs.
• Output can be marked as eligible for deletion once downloaded, or once a max age passes, whichever happens first. Other policies may be useful in the future, but this is a good policy to start.
• A periodic job should delete any outputs that are marked as eligible for deletion or orphaned.
• We can put the Redis behind a Python object with a defined interface, to facilitate future pluggability---i.e. adding hooks when jobs are complete, or plugging in other backends.
• Redis a good fit here because it can persist across restarts, but it's generally for ephemeral data. We don't want to be bothering with database migrations here; if we need a change in what's stored, the Redis should be blown away, not migrated.