Add support for `pre` and `post` scripts/commands

chevdor commented 11 months ago

A detailed description of the feature you would like to see added.

Sometimes, you need to perform a list of tasks and ensure you don't spam tasks too quickly. Or one may need to run tasks that depend on an external even to occur and create a file for instance.

This can also be considered as having to wait for a condition before executing a task. Such a condition would be "ensure this find call returns something" or "wait for sleep 1000" to return with a success status.

The idea would be to add a global option (overridable per group, overrridable per task) that would allow running a command pre or post running a task. This would apply to all tasks.

With pre set as sleep 10 for a group, the task would first run the pre script and wait for an exit. If the status is 0, we execute the task as it is done today. If the status is not 0, the task is aborted.

For the use case when the user needs to wait for an external event, a user script can be crafted to loop and wait for while and return 0 when the condition is met, or "give up" after a while and return an error status code.

Here are some examples: (I am using alias q=pueue)

q group add Greeter
for i in {1..10}; do 
   q add -g Greeter --pre 'sleep 10' --post 'echo Done' -- date
done

q group add GentleGreeter --pre 'sleep 10' --post 'echo Done'
for i in {1..10}; do 
   q add -g GentleGreeter --post 'echo Gentle' -- date
done

This would likely require a new state that could be called waiting or pre and post.

Explain your usecase of the requested feature

Consider for instance a scenario where you need to send thousands of curl request to an API and you want to ensure that you don't exceed whatever cap the targeted web service implements.

Alternatives

Currently, there would be options that are less elegant:

it would be possible to add a sleep <N> between each task but this bloats the tasks lists and the overviews
it would be possible to "wrap" the actual task into a script that prepend such a sleep N

Additional context

There is already the --after flag for the add command but it relates to other "single" tasks and not a task/script that would run before/after each task.

chevdor commented 11 months ago

I see https://github.com/Nukesor/pueue/wiki/Miscellaneous#aliases could help here as well

Nukesor commented 11 months ago

I thought a lot about this over the course of the last few days.

This feature would add a lot of complexity:

New executor logic based on groups and tasks.
Each task would need two additional commands to be executed. This adds loads of new edge-cases and questions.
- How should these be displayed in status. We're already running out of space.
- How should they be shown in the log output?
- Should logs be handled seperately or in the same log file?
- What happens if either of them fail? Should the whole task fail?
- ...

This change would add heaps of functionality, that would be better suited for a wrapper script, or a project with a better UI and the goal to provide a task scheduler/executor with advanced functionality.

Pueues goal always was and still is to be kind of minimalistic, but very good at what it does. This feature would go beyond that goal.

chevdor commented 11 months ago

I won't deny it adds complexity, we may take shortcuts if that helps make it doable:

How should these be displayed in status. We're already running out of space.

We could simply return the status code.

How should they be shown in the log output?

Status only ?

Should logs be handled seperately or in the same log file?

Those pre hooks are mainly gates to prevent or influence the scheduler. I think it would be fine to only consider their status.

On the other hand, why not letting the user decide and pre/post-pending the pre/post hooks to the regular log output ?

What happens if either of them fail? Should the whole task fail?

For the pre, 100% yes For post, I have no immediate answer... I would lean toward a 'yes' to keep it coherent with the pre

Nukesor commented 10 months ago

We could simply return the status code.

This will result in a bad UX, since users wouldn't know which part of their setup failed. I.e. was it the pre/post or the actual command that failed. This needs to be clearly communicated.

Status only ?

This is also critical. If anything in those scripts goes wrong, the user needs to have easy access to clearly visible output that allows them to debug any issues. The log output could be used for this, but it'll be tricky to get this right without making the output bloated or unintuitive.

For the pre, 100% yes For post, I have no immediate answer... I would lean toward a 'yes' to keep it coherent with the pre

If that's the case, what exactly keeps you from doing a simple && instead?

Your example from above would then look like this:

q group add Greeter
for i in {1..10}; do 
   q add -g Greeter 'sleep 10 && date  && echo Done` 
done

This is actually easier to type, shorter and more versatile, as you can utilize everything the shell offers.

Nukesor commented 9 months ago

Closing for now. I'll re-open if there's additional feedback.

Nukesor / pueue