search

alda-lang / alda-server-clj

A Clojure implementation of an Alda server
Other
2 stars 2 forks source link

Worker status pipeline for "alda parse" requests too #2

Closed daveyarwood closed 5 years ago

daveyarwood commented 7 years ago

Since version 1.0.0-rc42, we've had a worker status pipeline for alda play requests only, in order to more robustly handle worker requests that take a long time from the client side.

The idea is that when a worker receives a request, it starts handling it on a separate thread and goes into a particular working status like "parsing" or "playing," and it also immediately responds to the request to acknowledge that it received the request and is working on it. Then, the client can repeatedly make status requests to the worker so it can report when the worker is playing the score. The key thing here is that the client knows when to keep waiting vs. report that the request failed and exit.

We should extend this functionality to the alda parse command, too. Currently, the client just has a timeout interval, after which time it will retry the request again. If you do something like alda parse -c "(Thread/sleep 10000)" -m, the client will get impatient and keep retrying the request until every available worker is busy parsing the same score! This effectively kills all of the available workers because they all stop what they're doing and evaluate this score that makes them pause for 10 seconds, causing the server to think the workers are dead and evict them from the worker queue. Yikes!

The solution here is the same thing we implement for the alda play command, only the client would not print any of the status updates, since the alda parse command is intended to print only the parsed output to STDOUT.

daveyarwood commented 7 years ago

Should maybe rename the play-status command to worker-status.

daveyarwood commented 7 years ago

This is especially important for the new client/server implementation of the Alda REPL. Currently, long parse requests cause a worker to be blocked, which can hang the REPL unnecessarily (another available worker could be chosen instead of the blocked one).

jgkamat commented 6 years ago

There's a bug that's a symptom of this as well. If you parse a sufficiently large file it will return 'no workers found' instead of properly returning the parsed data (since it timesout)

bernardo-amorim commented 6 years ago

Is this still of interest? I'd love to give it try

daveyarwood commented 6 years ago

@bernardo-amorim Absolutely, help would be much appreciated! Let us know (either here or in the #development channel in Slack) if you need any help figuring it out.

daveyarwood commented 6 years ago

I think the first step is to extract the "job" logic from handle-code-play so that handle-code-parse can use it too: https://github.com/alda-lang/alda-server-clj/blob/34f5b4da625ad38f22c6ffa048a9239df1aa9223/src/alda/worker.clj#L121-L146

Then, run-job! will need to be modified to not be specific to playing, so it can handle parsing too: https://github.com/alda-lang/alda-server-clj/blob/34f5b4da625ad38f22c6ffa048a9239df1aa9223/src/alda/worker.clj#L85-L110

This could be as simple as taking an extra argument that is either :parse or :play and acting accordingly.

bernardo-amorim commented 6 years ago

OK, thanks for the response. I'll start coding it.

daveyarwood commented 5 years ago

I think this issue can be closed. I'm in the process of redesigning Alda for an eventual 2.0 release, and part of the new design is that most of the work will be done in the client, which will make this a non-issue.