Showing progress with 'salt' command

[kfdm]

One very minor advantage Ansible has for new users is that when you run a command, ansible-playbook it gives you progress about what steps are executing.

If you run the command on the minion side with salt-call, you can get some general output by adding -l info though it's a touch noisy if you don't know what you're looking for.

If you add state_events: True to your master configuration, then you can view the general progress by running salt-run state.events though this can also be a touch noisy.

If you run salt ... there is only the final report when everything is finished.

What would it take to be able to more easily watch the progress of a running state? Is this something that would make sense as an extension to the salt command or perhaps building it as a salt-run command would be better.

The local client provides a cmd_async command that returns a jobid which could be used to filter events

• salt/job/<JID>/new
• salt/job/<JID>/ret/<MID>
• salt/job/<JID>/prog/<MID>/<RUN NUM> though perhaps there is a slight delay between getting the jid from cmd_async and being able to subscribe to salt.modules.state.event

Are there perhaps any other race conditions that one would need to be aware of?

Copied from: https://groups.google.com/d/msgid/salt-users/afa89988-d358-40de-a451-9417bdb90206%40googlegroups.com.

[garethgreenaway]

@kfdm Thanks for the report. There is a progress out put module that can be used to show some progress during a Salt calls. It requires that the python progressbar library is installed where you're running Salt, eg. on the master. https://docs.saltstack.com/en/latest/ref/output/all/salt.output.progress.html#module-salt.output.progress

Real time output from the minion is tricky since they don't return their output to the master until the run has completed.

[kfdm]

--progress is one way to show that the command is still running, but I'm curious about what it would take to show more details about what is running. Unlike ansible the commands are not run in sync (some minions will finish before others) but for a proof-of-concept I'm not worried about that.

While it would require state_events: True to be set, my initial brainstorm was something similar to this

salt '*' state.apply
salt/job/<JID>/prog/A/1 file.manage /path/to/foo
salt/job/<JID>/prog/B/1 file.manage /path/to/foo
salt/job/<JID>/prog/A/2 service.running bar
salt/job/<JID>/prog/A/3 service.running baz
salt/job/<JID>/prog/B/2 service.running bar
...
Normal output.highstate

Perhaps I should try a quick implementation as a salt runner to test this idea a bit more.

[kedare]

I confirm this is more than needed. We have some huge highstate (and on Windows...) that can take between 30 minutes and 1h to apply, it's very frustrating to not have any progress report

[dbernadett]

I have the same need.

[Oloremo]

+1

[adam12b1]

+1! And I would say, even within the current architectural constraints on getting realtime data back from minions, it would still be better than nothing to at least have a "Now running [state] on [target]..." or similar update from the Salt master as it moves through its work... although, hm, I guess that's still currently only known on the minion, eh?

[Mapel88]

+1

[bmaeck]

+1

[pthirumalachar]

+1. + if this could report not only the state transitions, but could also capture the STD outs/warning of the underlying processes being run, is very much needed. And having the ability to list only the delta output would be very helpful

[digitalformula]

+1

[damon-atkins]

The minion is polled to see if the job is running. But it does not return where it is up to. state_events: True is most likely the best way. You can push event onto slack or similar as well.

[digitalformula]

The minion is polled to see if the job is running. But it does not return where it is up to. state_events: True is most likely the best way. You can push event onto slack or similar as well.

Out of interest - how difficult would it be to add some sort of progress bar? I understand that there are reasons why SaltStack doesn't do that now, and that's 100% fine, but if there was a command line switch that showed something as simple as "Running state 3/17", that would be huge.

salt-call state.apply blender.install --progress

My use case is the cloning of source from Github, which then needs to be built (Blender, specifically). I've got it all working without issue but there's zero output until the end, when the state application has either succeeded or failed.

[damon-atkins]

find_job response should lead to how many minions are working on it.

[tdonovic]

+1 States (when im testing) and cant see why the state times out until 20 min later is really slowing things up. Being able to tail the logs to the master that executed the state would make my life incredibly easy

[amunoz951]

+1 - we have salt-calls that can take hours and it's extremely helpful to see stdout in realtime

[BeehiveSystems]

+1 - we have salt-calls that can take hours and it's extremely helpful to see stdout in realtime

Same here, this would be a very useful feature.

[nsu700]

+1 , get used to the ansible progress output, and find it not safe enough to run a salt script without any progress and eventually throw me an error, killing me

[gigi206]

+1

[rawkode]

I'd love it if we could get real-time output when using salt-call --local.

Could this be implemented?

[klelifo]

+1

[damon-atkins]

find_job response should lead to how many minions are working on it. Maybe even return the number of states completed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Oloremo]

not stale

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[amunoz951]

Not stale

On Thu, Feb 6, 2020 at 8:39 AM stale[bot] notifications@github.com wrote:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

—

You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/46665?email_source=notifications&email_token=AEXQNJP44H7WH7KLWKS2C63RBQ4LDA5CNFSM4EW4AVY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK74M3I#issuecomment-582993517, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEXQNJOZH3LVVWGO5A2RFVTRBQ4LDANCNFSM4EW4AVYQ .

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[gigi206]

Not stale

[damon-atkins]

A solution would be logs per job. Instead of one main log. Or watching the event bus with a filter? Maybe a simple 1/123 tasks stored in the job file after each job is completed, with name of the last id executed. Maybe appead the state "id" to the job file as they are executed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Oloremo]

not stale. Event-based progress bar must be a thing

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[KevoSoftworks]

+1 We have a number of systems with high resource usage peaks Being able to monitor the progress of applying states would benefit our workflow during these periods.

[theunsa]

+1 fwiw

[hemantaltair]

+1 otherwise it's very difficult to go on a minion and check logs which is not practical when you have 1000 of infrastructure.

[SJay3]

+1 We need a progress for salt

[MalloZup]

+1

[network-shark]

+1

[alexkolomolo]

+999 This is really needed for anyone wanting to use salt seriously to manage their infra.

[amunoz951]

Our organization already abandoned salt in part for this reason. Reliability being the other.

On Fri, Dec 11, 2020 at 11:50 AM Kolomolo123 notifications@github.com wrote:

+999 This is really needed for anyone wanting to use salt seriously to manage their infra.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/46665#issuecomment-743393750, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEXQNJKDDFKFAHTAZX55HM3SUJZZPANCNFSM4EW4AVYQ .

[dmulyalin]

+1

Thoughts: might be possible to listen to events on master and write salt execution module in a way that logs will be emitted on event bus or fire events explicitly throughout module execution to indicate the progress. At the same time having a runner running on salt master to listen to this events or an output module that will be listening to such an events showing the progress.

IMHO, capability to subscribe to such progress events using Job ID sounds as useful feature as well.

[xyzst]

Would like to add my interest for this feature. I have a long running job on a minion, and I have to manually check the minion itself if the job has stalled or not.

[auphofBSF]

I Have observed in minion tail target-logs -f and journalctl -f rich sources of logging that could be useful in providing a more automated progress feedback., is there any work current or in planning at Saltstack to provide some progress reporting in running a particular salt state

I often find my salt-ssh <target> state.apply <some state> failing with what seems like an ssh timeout. however I have been able to monitor the to see the state.apply is still functioning, utilizing logs and journalctl

The following is a typical response I get , which is not very helpful

# Response from salt-ssh during a long running state
[ERROR   ] JSON Render failed for:
Connection to <target> closed by remote host.
[ERROR   ] Expecting value: line 1 column 1 (char 0)

[sticky-note]

+ 1 to have this kind of real time reporting feature in SaltStack

[raqua]

This is needed!

[Jibun-no-Kage]

THIS IS NEEDED!

[oijkn]

I use Gitlab-ci to deploy projects with Saltstack but effectively the real time events is missing. When the state.apply finished, then the full report appears.

Can you add this feature please?

[damon-atkins]

The first step is the salt-minion to record more information in the job file. I think it would need a SEP.

[damon-atkins]

What information do people want? X/28 tasks complete Or List of all task ID and status e.g. Running/InProgress, Stable/Same/Identical, Changed/Mocked, Skipped, Failed

I do not think it should output the full data, however if it did then the user could control how much is displayed. However it would increase overhead.

[alexkolomolo]

X/28 tasks complete would be a great!!

On Sun, Nov 27, 2022, at 2:24 AM, Damon Atkins wrote:

What information do people want? X/28 tasks complete Or List of all task ID and status e.g. Running/InProgress, Stable/Same/Identical, Changed/Mocked, Skipped, Failed

I do not think it should output the result, it should be suitable for an interface.

— Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/46665#issuecomment-1328147471, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGQGDKBA6NZMAPUZIVJEEP3WKKZ4TANCNFSM4EW4AVYQ. You are receiving this because you commented.Message ID: @.***>