pueue start --stashed

[alsuren]

A detailed description of the feature you would like to see added.

I would like to be able to resume all stashed tasks without knowing their ids

Currently, it looks like this:

$ pueue add --stashed true
New task added (id 10).

$ pueue start
Group "default" is being resumed.

$ pueue status
Group "default" (1 parallel): running
───────────────────────────────────────────────────────────────────────────
 Id   Status    Command   Path                         Start      End      
═══════════════════════════════════════════════════════════════════════════
...
───────────────────────────────────────────────────────────────────────────
 10   Stashed   true      /Users/david
───────────────────────────────────────────────────────────────────────────

I have to find out and supply the id in order to unstash it

$ pueue start 10
Tasks are being started: 10

$ pueue status
Group "default" (1 parallel): running
───────────────────────────────────────────────────────────────────────────
 Id   Status    Command   Path                         Start      End      
═══════════════════════════════════════════════════════════════════════════
...
───────────────────────────────────────────────────────────────────────────
 10   Success   true      /Users/david                 18:28:39   18:28:39 
───────────────────────────────────────────────────────────────────────────

Really I would like to be able to do

$ pueue start --stashed

to unstash all stashed tasks. Note that some careful handling of delayed tasks is required, because they are also encoded as stashed tasks.

Explain your usecase of the requested feature

I stumbled across this because I had started a long-running task in my terminal (rebuilding docker containers and importing a db snapshot) and I wanted to bring up my dev server when it was finished. I didn't really want to run either of the tasks under pueue's supervision. I was basically using pueue as a glorified semaphore system.

What I wanted to do was run this in one terminal:

pueue wait `pueue add --stashed --print-task-id true` && yarn start

and then blindly type pueue start into the terminal of the long-running process (I knew that the process doesn't read from stdin so this would be run by bash as the next command).

Alternatives

a) I could write a wrapper around pueue + jq to find the id of the stashed processes and start them, but that requires forethought (which is how I got myself into this situation in the first place). Typing it in an editor and pasting probably wouldn't work because bash would recognise it as quoted paste and wait for me to hit enter again.

b) after a bit of experimentation I found that this kind-of works, and allows me to type pueue start in the other terminal as I intended, to unpause the task:

id=`pueue add -p sleep 2` && sleep 1 && pueue pause $id && pueue wait $id && yarn start

It is inherently racy though. If I remove the sleep 1, or if the default group is already at max parallel limit then it will fail with The command failed for tasks: 17

c) if pueue start had a --paused option then this would also work, but I suspect that skipping straight to paused might break the state machine a bit, so probably isn't a good idea.

d) if pueue had a way to adopt processes then I could potentially run something like

pueue add --adopt-pid `pgrep -f my-long-running-script.sh` && pueue wait

This requires me to know a pgrep query that will find my long running process without accidentally catching any others in the net.

e) if I knew of a tool that could wait for a process then I could potentially use that instead. Probably wouldn't be that difficult to write either

f) I could potentially have run ctrl+z and then fg && yarn start, but then I would have to shuffle my terminal windows around, because I'd started the long running task in a different place from where I usually put my dev server.

g) potentially pueue start --all could be changed to also start stashed tasks, but this change of behaviour might be surprising to experienced users

h) potentially an --all or --stashed flag could be added to pueue enqueue instead/as well. Care would be needed to explain the discrepancy if pueue enqueue --all does a different thing to pueue start --all.

Additional context

No response

[Nukesor]

Sorry for the long delay.

I went through your comment in detail and to be honest, I'm not sure your usecase fits into Pueue's feature scope :sweat_smile: . Using pueue as a semaphore system for tasks that aren't even executed by pueue isn't really something I want to enable by adding additional features.

Usually, something like this would be either handled via normal tasks and dependencies or via groups. Still, using Pueue for your usecase is a bit of an overkill.

I would propose to use something like this:

block_and_wait() {
    if [ "$#" -eq 0  ]; then
        echo "First parameter should be the name of the semaphore."
        return 1;
    fi
    mkdir "$XDG_RUNTIME_DIR/semaphores"

    if [[ -f "$XDG_RUNTIME_DIR/semaphores/$1" ]]; then
        echo "Semaphore $1 already exists."
    else
        echo "Creating semaphore $1"
        touch "$XDG_RUNTIME_DIR/semaphores/$1"
    fi

    # Wait until the semaphore has been removed
    while true; do
        if [[ ! -f "$XDG_RUNTIME_DIR/semaphores/$1" ]]; then
            return 0;
        fi
        sleep 2
    done
}

unblock() {
    if [ "$#" -eq 0 ]; then
        echo "First parameter should be the name of the semaphore."
        return 1;
    fi

    if [[ -f "$XDG_RUNTIME_DIR/semaphores/$1" ]]; then
        echo "Removing semaphore $1"
        rm "$XDG_RUNTIME_DIR/semaphores/$1"
    else
        echo "Couldn't find semaphore at $XDG_RUNTIME_DIR/semaphores/$1"
        return 1;
    fi
}

This allows you to call some long running function in any terminal.

Afterwards you go ahead and run one or multiple follow-up calls in other terminals like this:

block_and_wait rofl && command_1

Another terminal:

block_and_wait rofl && command_2

Then you run the following in your first terminal (your docker rebuild):

unblock rofl

[Nukesor]

I actually added those two functions to my dotfiles https://github.com/Nukesor/dotfiles/commit/e9f2c13ce31ac47dd5c09d21229e01c1d7af94ed. They seem quite handy.

[Nukesor]

I'm moving this to Closed for now. Feel free to comment and I'll re-open if necessary :)