salt-run state.orch ceph.stage.0 - mines failure

noelmcloughlin commented 6 years ago

Description of Issue/Question

Running export DEV_ENV=true; sudo salt-run state.orch ceph.stage.0 gets a mines state failure.

How can I troubleshoot and resolve this issue / any pointers?

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

$ sudo fdisk  -l
Disk /dev/sdb: 375 GiB, 402653184000 bytes, 98304000 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xc654ff28

Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1  *     2048 209715166 209713119  100G 83 Linux

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

$ sudo salt-run state.orch ceph.stage.0
.... some boto noise ....
[WARNING ] All minions are ready
[ERROR   ] {u'ret': {u'masterminion': {u'module_|-manage_salt_minion_for_mines_|-mine.update_|-run': {u'comment': u'Module function mine.update executed', u'name': u'mine.update', u'start_time': u'19:34:53.091793', u'result': True, u'duration': 1.166, u'__run_num__': 2, u'__sls__': u'ceph.mines.default', u'changes': {u'ret': None}, u'__id__': u'manage_salt_minion_for_mines'}, u'module_|-add_mine_cephdisks.list_to_minion_|-mine.send_|-run': {u'comment': u'Module function mine.send executed', u'name': u'mine.send', u'start_time': u'19:34:53.082652', u'result': False, u'duration': 8.699, u'__run_num__': 1, u'__sls__': u'ceph.mines.default', u'changes': {u'ret': False}, u'__id__': u'add_mine_cephdisks.list_to_minion'}, u'file_|-configure_mine_functions_conf_|-/etc/salt/minion.d/mine_functions.conf_|-managed': {u'comment': u'File /etc/salt/minion.d/mine_functions.conf updated', u'pchanges': {}, u'name': u'/etc/salt/minion.d/mine_functions.conf', u'start_time': u'19:34:53.015295', u'result': True, u'duration': 52.565, u'__run_num__': 0, u'__sls__': u'ceph.mines.default', u'changes': {u'diff': u'New file', u'mode': u'0644'}, u'__id__': u'configure_mine_functions_conf'}}}, u'out': u'highstate'}
masterminion_master:
  Name: sync master - Function: salt.state - Result: Changed Started: - 19:33:24.154135 Duration: 881.159 ms
  Name: salt-api - Function: salt.state - Result: Changed Started: - 19:33:25.035489 Duration: 1969.384 ms
  Name: repo master - Function: salt.state - Result: Clean Started: - 19:33:27.005024 Duration: 619.675 ms
  Name: metapackage master - Function: salt.state - Result: Clean Started: - 19:33:27.624874 Duration: 461.204 ms
  Name: prepare master - Function: salt.state - Result: Changed Started: - 19:33:28.086268 Duration: 72235.469 ms
  Name: filequeue.remove - Function: salt.runner - Result: Changed Started: - 19:34:40.321899 Duration: 1762.72 ms
  Name: filequeue.enqueue - Function: salt.runner - Result: Changed Started: - 19:34:42.084774 Duration: 369.848 ms
  Name: minions.ready - Function: salt.runner - Result: Changed Started: - 19:34:42.454792 Duration: 574.155 ms
  Name: repo - Function: salt.state - Result: Clean Started: - 19:34:43.029160 Duration: 574.728 ms
  Name: metapackage minions - Function: salt.state - Result: Clean Started: - 19:34:43.604091 Duration: 434.69 ms
  Name: common packages - Function: salt.state - Result: Changed Started: - 19:34:44.038956 Duration: 7659.407 ms
  Name: sync - Function: salt.state - Result: Changed Started: - 19:34:51.698568 Duration: 853.937 ms
----------
          ID: mines
    Function: salt.state
      Result: False
     Comment: Run failed on minions: masterminion
     Started: 19:34:52.552653
    Duration: 566.53 ms
     Changes:   
              masterminion:
                Name: /etc/salt/minion.d/mine_functions.conf - Function: file.managed - Result: Changed Started: - 19:34:53.015295 Duration: 52.565 ms
              ----------
                        ID: add_mine_cephdisks.list_to_minion
                  Function: module.run
                      Name: mine.send
                    Result: False
                   Comment: Module function mine.send executed
                   Started: 19:34:53.082652
                  Duration: 8.699 ms
                   Changes:   
                            ----------
                            ret:
                                False
                Name: mine.update - Function: module.run - Result: Changed Started: - 19:34:53.091793 Duration: 1.166 ms

              Summary for masterminion
              ------------
              Succeeded: 2 (changed=3)
              Failed:    1
              ------------
              Total states run:     3
              Total run time:  62.430 ms
  Name: updates - Function: salt.state - Result: Clean Started: - 19:34:53.119383 Duration: 5478.616 ms
  Name: mds restart noop - Function: test.nop - Result: Clean Started: - 19:34:58.598181 Duration: 0.302 ms

Versions Report

0.9.5+62.gac468eb

ii salt-api 2018.3.1+ds-1 all Generic, modular network access system ii salt-common 2018.3.1+ds-1 all shared libraries that salt requires for all packages ii salt-master 2018.3.1+ds-1 all remote manager to administer servers via salt ii salt-minion 2018.3.1+ds-1 all client package for salt, the distributed remote execution system

swiftgist commented 6 years ago

Of the problems that I have experienced with mines, I do not believe mine.send has been one. :)

Since it's a module, you can run it directly

`# salt 'minionX*' mine.send cephdisks.list

The mine.send is there to prevent having to restart the Salt minion to recognize the new configuration file. We are using the Salt mine to cache cephdisks.list since that calls hwinfo or lshw which can take minutes on real hardware.

As far as debugging, I think you are at the enable "log_level: debug" on that minion, restart the minion and then try the command above step. Check /var/log/salt/minion for explicit messages.

lordfolken commented 5 years ago

I had this issue on debian. I found in the minion log that i was missing lshw and hwinfo and gdisk packages:

2019-01-15 21:43:25,383 [salt.loaded.int.module.mine:169 ][ERROR ][13348] Function cephdisks.list in mine_functions failed to execute 2019-01-15 21:43:25,432 [salt.loaded.int.module.mine:169 ][ERROR ][13348] Function cephimages.list in mine_functions failed to execute 2019-01-15 21:44:11,607 [salt.loader.salt-master.sigkill.noexit.ext.module.cephdisks:290 ][ERROR ][13585] Can not find a proper hardware detection tool. Install lshw or hwinfo in order to retrive hardware information

jschmid1 commented 5 years ago

Install lshw or hwinfo in order to retrive hardware information ^ ^ @lordfolken Do you have one of them installed?

SUSE / DeepSea