SUSE / DeepSea

A collection of Salt files for deploying, managing and automating Ceph.
GNU General Public License v3.0
160 stars 75 forks source link

[next] Refine the runner function decorator #1785

Closed jschmid1 closed 4 years ago

jschmid1 commented 4 years ago

This will not only handle the Exception case but also return the correct values based on the context (orchestrator, runner)

This allows us to have consistent behavior independent of whether we are calling a runner or a module underneath.

We have three modes:

human (default)

We're returning the aggregated results in a human readable form (success/failure)


orchestrator

We're returning a datastructure of our choice (current this returns a list of returnvalues, which is not terribly insightful). We can adapt this returnvalue anytime when we have real requirements from the orchestrator.


runner

That's for internal uses only. This will re-raise the Exception in form of a str when an isssue is encountered. If it succeeds it will currently just print an aggregated list of returnvalues.

click below to see all possible cases

click me ``` admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.good keyring.mon on admin: success keyring.mon on mon3: success keyring.mon on mon1: success keyring.mon on mon2: success success admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.good called_by_orch=True keyring.mon on admin: success keyring.mon on mon1: success keyring.mon on mon3: success keyring.mon on mon2: success - True - True admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.bad keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' Caught an Exception while running .(ment for a human) failure admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.bad called_by_orch=True keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' Caught an Exception while running .(ment for orchestrator) admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.bad called_by_runner=True keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' Exception occurred in runner test.bad: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/client/mixins.py", line 377, in low data['return'] = func(*args, **kwargs) File "/srv/modules/runners/ext_lib/decorators.py", line 39, in newfunc results = f(*a, **kw) File "/srv/modules/runners/test.py", line 101, in bad arguments=['admin']) File "/srv/modules/runners/ext_lib/operation.py", line 22, in exec_module raise ModuleException(False, ret) ext_lib.exceptions.ModuleException: Insert some nice data from self.data : False admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.good called_by_runner=True keyring.mon on admin: success keyring.mon on mon2: success keyring.mon on mon3: success keyring.mon on mon1: success - True - True admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.runner_calls_runner keyring.mon on admin: success keyring.mon on mon3: success keyring.mon on mon2: success keyring.mon on mon1: success keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' Caught an Exception in runner test.bad (ment for a human) failure admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.runner_calls_runner called_by_orch=True keyring.mon on admin: success keyring.mon on mon2: success keyring.mon on mon3: success keyring.mon on mon1: success keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' Caught an Exception in runner test.bad (ment for orchestrator) admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.runner_calls_runner called_by_runner=True keyring.mon on admin: success keyring.mon on mon2: success keyring.mon on mon1: success keyring.mon on mon3: success keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' Exception occurred in runner test.runner_calls_runner: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/client/mixins.py", line 377, in low data['return'] = func(*args, **kwargs) File "/srv/modules/runners/ext_lib/decorators.py", line 39, in newfunc results = f(*a, **kw) File "/srv/modules/runners/test.py", line 123, in runner_calls_runner result, data = exec_runner('test.bad') File "/srv/modules/runners/ext_lib/utils.py", line 237, in exec_runner raise RunnerException(cmd) ext_lib.exceptions.RunnerException: test.bad admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.runner_calls_runner_calls_runner keyring.mon on admin: success keyring.mon on mon1: success keyring.mon on mon3: success keyring.mon on mon2: success keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' Caught an Exception in runner test.runner_calls_runner (ment for a human) failure admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.runner_calls_runner_calls_runner called_by_orch=True keyring.mon on admin: success keyring.mon on mon3: success keyring.mon on mon1: success keyring.mon on mon2: success keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' Caught an Exception in runner test.runner_calls_runner (ment for orchestrator) admin:/srv/salt/_modules/ext_lib/c-d # salt-run test.runner_calls_runner_calls_runner called_by_runner=True keyring.mon on admin: success keyring.mon on mon2: success keyring.mon on mon1: success keyring.mon on mon3: success keyring.mon_failure on mon2: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon3: failure No such file or directory: 'ceph-daemon' keyring.mon_failure on mon1: failure No such file or directory: 'ceph-daemon' Exception occurred in runner test.runner_calls_runner_calls_runner: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/client/mixins.py", line 377, in low data['return'] = func(*args, **kwargs) File "/srv/modules/runners/ext_lib/decorators.py", line 39, in newfunc results = f(*a, **kw) File "/srv/modules/runners/test.py", line 133, in runner_calls_runner_calls_runner result, data = exec_runner('test.runner_calls_runner') File "/srv/modules/runners/ext_lib/utils.py", line 237, in exec_runner raise RunnerException(cmd) ext_lib.exceptions.RunnerException: test.runner_calls_runner ```

Signed-off-by: Joshua Schmid jschmid@suse.de

swiftgist commented 4 years ago

Honestly, I would stick with the original decorator from ceph-volume https://github.com/ceph/ceph/blob/master/src/ceph-volume/ceph_volume/decorators.py. For our use case with Salt, I can see removing the exit=True logic, since sys.exit(1) is not graceful in modules and runners.

With respect to the modes, I would consider those results. We originally started with human=True and machine=False in several examples, but using out=yaml simplifies the interaction with modules and runners. This also gives control to the caller. We do have modules specifically for returning data unmodified in a Salt state, for example. Passing additional parameters to decorators to handle these cases complicates the decorator interface. Let this decorator handle exceptions and nothing more.

jschmid1 commented 4 years ago

I think we still have three distinct outputs that can either be defined with out=<output_type> or output_type=<bool>

Personally I don't mind either solution. While the former solution will reduce the number of kwargs to one, the latter is more explicite.

jschmid1 commented 4 years ago

Honestly, I would stick with the original decorator from ceph-volume https://github.com/ceph/ceph/blob/master/src/ceph-volume/ceph_volume/decorators.py. For our use case with Salt, I can see removing the exit=True logic, since sys.exit(1) is not graceful in modules and runners.

While I agree that the logic for when to return what doesn't necessarily belong in the decorator, we can't just move it out (edit: easily) . If you have a better place to put it, feel free to suggest it.