badfish.py doesn't clear job queue

bengland2 commented 5 years ago

I can't seem to clear the job queue with either badfish or DRAC GUI, this is preventing me from changing the boot order.

[bengland@bene-laptop badfish]$ ./badfish.py -H mgmt-e23-h23-740xd -u quads -p 494292 -i config/idrac_interfaces.yml --clear-jobs
- INFO     - Systems service: /redfish/v1/Systems/System.Embedded.1.
- INFO     - Managers service: /redfish/v1/Managers/iDRAC.Embedded.1.
- WARNING  - iDRAC version installed does not support DellJobService
- WARNING  - Clearing job queue for job IDs: ['JID_640133055279'].
- ERROR    - Job queue not cleared, current job queue contains jobs: ['JID_640133055279'].

bengland2 commented 5 years ago

Chris Enright seems to know why this happens, apparently JID_CLEARALL doesn't work on 740xd? @QuantumPosix ?

QuantumPosix commented 5 years ago

I am testing this now, it did not earlier while a job was there, and took a "jobqueue delete --all" instead of "jobqueue delete -i JID_CLEARALL". I will be testing further on a node that is not involved in any reservations. ~ Thanks,

Chris 978.392.1043

On Thu, Jul 25, 2019 at 10:57 AM Ben England notifications@github.com wrote:

Chris Enright seems to know why this happens, apparently JID_CLEARALL doesn't work on 740xd? @QuantumPosix https://github.com/QuantumPosix ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-performance/badfish/issues/45?email_source=notifications&email_token=AG3IDEEZPRSJ3HT3IRSIVHDQBG5NHA5CNFSM4IGULVX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2ZXTZQ#issuecomment-515078630, or mute the thread https://github.com/notifications/unsubscribe-auth/AG3IDEDNR7CNNDXTPHY2BD3QBG5NHANCNFSM4IGULVXQ .

QuantumPosix commented 5 years ago

So it looks like a firmware update for iDrac9 will fix the issue as annotated below:

Old iDrac Firmware(3.21.21.21):

  ./badfish.py -H $host -u root -p "calvin" --clear-jobs
    - INFO     - Systems service: /redfish/v1/Systems/System.Embedded.1.
    - INFO     - Managers service: /redfish/v1/Managers/iDRAC.Embedded.1.
    - WARNING  - iDRAC version installed does not support DellJobService
    - WARNING  - Clearing job queue for job IDs: ['JID_641574909915'].
    - ERROR    - Job queue not cleared, current job queue contains jobs: ['JID_641574909915'].

New iDrac Firmware(3.34.34.34):

 ./badfish.py -H $host -u root -p "calvin" --clear-jobs
    - INFO     - Systems service: /redfish/v1/Systems/System.Embedded.1.
    - INFO     - Managers service: /redfish/v1/Managers/iDRAC.Embedded.1.
    - INFO     - Job queue for iDRAC $host successfully cleared.

Same job in the queue - seems to be a minimum iDrac version required for iDrac9 machines as well. I will start updating the firmware on the hosts.

grafuls commented 5 years ago

So it looks like a firmware update for iDrac9 will fix the issue as annotated below:

Old iDrac Firmware(3.21.21.21):

./badfish.py -H $host -u root -p "calvin" --clear-jobs
  - INFO     - Systems service: /redfish/v1/Systems/System.Embedded.1.
  - INFO     - Managers service: /redfish/v1/Managers/iDRAC.Embedded.1.
  - WARNING  - iDRAC version installed does not support DellJobService
  - WARNING  - Clearing job queue for job IDs: ['JID_641574909915'].
  - ERROR    - Job queue not cleared, current job queue contains jobs: ['JID_641574909915'].

New iDrac Firmware(3.34.34.34):

./badfish.py -H $host -u root -p "calvin" --clear-jobs
 - INFO     - Systems service: /redfish/v1/Systems/System.Embedded.1.
 - INFO     - Managers service: /redfish/v1/Managers/iDRAC.Embedded.1.
 - INFO     - Job queue for iDRAC $host successfully cleared.

Same job in the queue - seems to be a minimum iDrac version required for iDrac9 machines as well. I will start updating the firmware on the hosts.

Closing as per last findings from @QuantumPosix.

redhat-performance / badfish

badfish.py doesn't clear job queue #45