search

dell / iDRAC-Redfish-Scripting

Python and PowerShell scripting for Dell EMC PowerEdge iDRAC REST API with DMTF Redfish
GNU General Public License v2.0
600 stars 278 forks source link

Handling the Firmware Update Dependency by using Lifecycle Controller using Redfish API #70

Closed aviv12825 closed 4 years ago

aviv12825 commented 5 years ago

Hi,

Any option to use Redfish API for Handling the Firmware Update Dependency by using Lifecycle Controller 3 ? like the picture below.

image

texroemer commented 5 years ago

Hi @aviv12825

Supported in iDRAC 9 3.30, you can now perform repository updates using Redfish with OEM extension "redfish/v1/Dell/Systems/System.Embedded.1/DellSoftwareInstallationService/Actions/DellSoftwareInstallationService.InstallFromRepository".

Example below shows executing script "InstallFromRepositoryREDFISH.py" which will first use to the script to verify what packages are available to update from the repository. This is optional to do but recommended if you first want to check the repository to see what devices and versions are listed before applying updates. Second i will then apply the updates which will update BIOS, iDRAC and storage controller firmware.

C:\Python27>InstallFromRepositoryREDFISH.py -ip 192.168.0.120 -u root -p calvin -i y --ipaddress 192.168.0.130 --sharetype NFS --sharename nfs_share_vm/R940_repo --applyupdate False

ShareType: NFS IPAddress: 192.168.0.130 ApplyUpdate: False ShareName: nfs_share_vm/R940_repo

@odata.type: #DellJob.v1_0_1.DellJob JobState: Completed Description: Job Instance TargetSettingsURI: None @odata.id: /redfish/v1/Managers/iDRAC.Embedded.1/Jobs/JID_608789389397 @odata.context: /redfish/v1/$metadata#DellJob.DellJob MessageArgs: [u'NA'] CompletionTime: 2019-06-18T12:29:05 PercentComplete: 100 StartTime: TIME_NOW MessageId: RED001 Message: Job completed successfully. EndTime: None Id: JID_608789389397 JobType: RepositoryUpdate Name: Repository Update

C:\Python27>InstallFromRepositoryREDFISH.py -ip 192.168.0.120 -u root -p calvin -r y

-PASS: POST command passed to get repo update list, status code 200 returned

<?xml version="1.0"?>
<CIM xmlns:fo="http://www.w3.org/1999/XSL/Format" CIMVERSION="2.0" DTDVERSION="2.0">
  <MESSAGE ID="4711" PROTOCOLVERSION="1.0">
    <SIMPLEREQ>
      <VALUE.NAMEDINSTANCE>
        <INSTANCENAME CLASSNAME="DCIM_RepoUpdateSWID"><PROPERTY NAME="Criticality" TYPE="string"><VALUE>1</VALUE></PROPERTY><PROPERTY NAME="DisplayName" TYPE="string"><VALUE>PERC 10.3 RAID Controller Firmware, 50.5.1-2633, A07</VALUE></PROPERTY><PROPERTY NAME="BaseLocation" TYPE="string"><VALUE/></PROPERTY><PROPERTY NAME="PackagePath" TYPE="string"><VALUE>SAS-RAID_Firmware_FJHK3_WN64_50.5.1-2633_A07.EXE</VALUE></PROPERTY><PROPERTY NAME="PackageName" TYPE="string"><VALUE>SAS-RAID_Firmware_FJHK3_WN64_50.5.1-2633_A07.EXE</VALUE></PROPERTY><PROPERTY NAME="PackageVersion" TYPE="string"><VALUE>50.5.1-2633</VALUE></PROPERTY><PROPERTY NAME="RebootType" TYPE="string"><VALUE>HOST</VALUE></PROPERTY><PROPERTY NAME="JobID" TYPE="string"><VALUE/></PROPERTY>
    <PROPERTY NAME="Target" TYPE="string"><VALUE>DCIM:INSTALLED#301_C_RAID.Slot.1-1</VALUE></PROPERTY><PROPERTY NAME="ComponentID" TYPE="string"><VALUE>104893</VALUE></PROPERTY><PROPERTY NAME="ComponentType" TYPE="string"><VALUE>FRMW</VALUE></PROPERTY><PROPERTY.ARRAY NAME="ComponentInfoValue" TYPE="string"><VALUE.ARRAY><VALUE>1000:0016:1028:1FCB</VALUE></VALUE.ARRAY></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInfoName" TYPE="string"><VALUE.ARRAY><VALUE>VendorID:DeviceID:SubVendorID:SubDeviceID</VALUE></VALUE.ARRAY></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInfoTarget" TYPE="string"><VALUE.ARRAY><VALUE>DCIM:INSTALLED#301_C_RAID.Slot.1-1</VALUE></VALUE.ARRAY></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInstalledVersion" TYPE="string"><VALUE.ARRAY><VALUE>50.5.0-1750</VALUE></VALUE.ARRAY></PROPERTY.ARRAY>

  </INSTANCENAME>
      </VALUE.NAMEDINSTANCE>
      <VALUE.NAMEDINSTANCE>
        <INSTANCENAME CLASSNAME="DCIM_RepoUpdateSWID"><PROPERTY NAME="Criticality" TYPE="string"><VALUE>1</VALUE></PROPERTY><PROPERTY NAME="DisplayName" TYPE="string"><VALUE>Dell EMC Server PowerEdge BIOS R740/R740xd/R640/R940/7920R Version 2.2.10</VALUE></PROPERTY><PROPERTY NAME="BaseLocation" TYPE="string"><VALUE/></PROPERTY><PROPERTY NAME="PackagePath" TYPE="string"><VALUE>BIOS_WG5H6_WN64_2.2.10.EXE</VALUE></PROPERTY><PROPERTY NAME="PackageName" TYPE="string"><VALUE>BIOS_WG5H6_WN64_2.2.10.EXE</VALUE></PROPERTY><PROPERTY NAME="PackageVersion" TYPE="string"><VALUE>2.2.10</VALUE></PROPERTY><PROPERTY NAME="RebootType" TYPE="string"><VALUE>HOST</VALUE></PROPERTY><PROPERTY NAME="JobID" TYPE="string"><VALUE/></PROPERTY>
    <PROPERTY NAME="Target" TYPE="string"><VALUE>DCIM:INSTALLED#741__BIOS.Setup.1-1</VALUE></PROPERTY><PROPERTY NAME="ComponentID" TYPE="string"><VALUE>159</VALUE></PROPERTY><PROPERTY NAME="ComponentType" TYPE="string"><VALUE>BIOS</VALUE></PROPERTY><PROPERTY.ARRAY NAME="ComponentInfoValue" TYPE="string"><VALUE.ARRAY/></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInfoName" TYPE="string"><VALUE.ARRAY/></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInfoTarget" TYPE="string"><VALUE.ARRAY><VALUE>DCIM:INSTALLED#741__BIOS.Setup.1-1</VALUE></VALUE.ARRAY></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInstalledVersion" TYPE="string"><VALUE.ARRAY><VALUE>1.6.12</VALUE></VALUE.ARRAY></PROPERTY.ARRAY>

  </INSTANCENAME>
      </VALUE.NAMEDINSTANCE>
      <VALUE.NAMEDINSTANCE>
        <INSTANCENAME CLASSNAME="DCIM_RepoUpdateSWID"><PROPERTY NAME="Criticality" TYPE="string"><VALUE>1</VALUE></PROPERTY><PROPERTY NAME="DisplayName" TYPE="string"><VALUE>iDRAC with Lifecycle Controller,3.32.32.32, A00</VALUE></PROPERTY><PROPERTY NAME="BaseLocation" TYPE="string"><VALUE/></PROPERTY><PROPERTY NAME="PackagePath" TYPE="string"><VALUE>iDRAC-with-Lifecycle-Controller_Firmware_VRYKT_WN64_3.32.32.32_A00.EXE</VALUE></PROPERTY><PROPERTY NAME="PackageName" TYPE="string"><VALUE>iDRAC-with-Lifecycle-Controller_Firmware_VRYKT_WN64_3.32.32.32_A00.EXE</VALUE></PROPERTY><PROPERTY NAME="PackageVersion" TYPE="string"><VALUE>3.32.32.32</VALUE></PROPERTY><PROPERTY NAME="RebootType" TYPE="string"><VALUE>IDRAC</VALUE></PROPERTY><PROPERTY NAME="JobID" TYPE="string"><VALUE/></PROPERTY>
    <PROPERTY NAME="Target" TYPE="string"><VALUE>DCIM:INSTALLED#iDRAC.Embedded.1-1#IDRACinfo</VALUE></PROPERTY><PROPERTY NAME="ComponentID" TYPE="string"><VALUE>25227</VALUE></PROPERTY><PROPERTY NAME="ComponentType" TYPE="string"><VALUE>FRMW</VALUE></PROPERTY><PROPERTY.ARRAY NAME="ComponentInfoValue" TYPE="string"><VALUE.ARRAY/></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInfoName" TYPE="string"><VALUE.ARRAY/></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInfoTarget" TYPE="string"><VALUE.ARRAY><VALUE>DCIM:INSTALLED#iDRAC.Embedded.1-1#IDRACinfo</VALUE></VALUE.ARRAY></PROPERTY.ARRAY><PROPERTY.ARRAY NAME="ComponentInstalledVersion" TYPE="string"><VALUE.ARRAY><VALUE>3.30.30.30</VALUE></VALUE.ARRAY></PROPERTY.ARRAY>

  </INSTANCENAME>
      </VALUE.NAMEDINSTANCE>
    </SIMPLEREQ>
  </MESSAGE>
</CIM>

C:\Python27>InstallFromRepositoryREDFISH.py -ip 192.168.0.120 -u root -p calvin -i y --ipaddress 192.168.0.130 --sharetype NFS --sharename nfs_share_vm/R940_repo --applyupdate True --rebootneeded True

ShareType: NFS RebootNeeded: True ApplyUpdate: True ShareName: nfs_share_vm/R940_repo IPAddress: 192.168.0.130

@odata.type: #DellJob.v1_0_1.DellJob JobState: Completed Description: Job Instance TargetSettingsURI: None @odata.id: /redfish/v1/Managers/iDRAC.Embedded.1/Jobs/JID_608892918527 @odata.context: /redfish/v1/$metadata#DellJob.DellJob MessageArgs: [u'NA'] CompletionTime: 2019-06-18T15:33:57 PercentComplete: 100 StartTime: TIME_NOW MessageId: RED001 Message: Job completed successfully. EndTime: None Id: JID_608892918527 JobType: RepositoryUpdate Name: Repository Update

@odata.type: #DellJob.v1_0_1.DellJob JobState: Completed Description: Job Instance TargetSettingsURI: None @odata.id: /redfish/v1/Managers/iDRAC.Embedded.1/Jobs/JID_608892983799 @odata.context: /redfish/v1/$metadata#DellJob.DellJob MessageArgs: [] CompletionTime: 2019-06-18T15:33:54 PercentComplete: 100 StartTime: TIME_NOW MessageId: PR19 Message: The specified job has completed successfully. EndTime: TIME_NA Id: JID_608892983799 JobType: FirmwareUpdate Name: update:DCIM:INSTALLED#741__BIOS.Setup.1-1

@odata.type: #DellJob.v1_0_1.DellJob JobState: Completed Description: Job Instance TargetSettingsURI: None @odata.id: /redfish/v1/Managers/iDRAC.Embedded.1/Jobs/JID_608893194634 @odata.context: /redfish/v1/$metadata#DellJob.DellJob MessageArgs: [] CompletionTime: 2019-06-18T15:33:54 PercentComplete: 100 StartTime: TIME_NOW MessageId: PR19 Message: Job completed successfully. EndTime: TIME_NA Id: JID_608893194634 JobType: FirmwareUpdate Name: update:DCIM:INSTALLED#301_C_RAID.Slot.1-1

@odata.type: #DellJob.v1_0_1.DellJob JobState: Completed Description: Job Instance TargetSettingsURI: None @odata.id: /redfish/v1/Managers/iDRAC.Embedded.1/Jobs/JID_608900371405 @odata.context: /redfish/v1/$metadata#DellJob.DellJob MessageArgs: [] CompletionTime: 2019-06-18T15:36:19 PercentComplete: 100 StartTime: TIME_NOW MessageId: RED001 Message: Job completed successfully. EndTime: None Id: JID_608900371405 JobType: FirmwareUpdate Name: update:DCIM:INSTALLED#iDRAC.Embedded.1-1#IDRACinfo

tyagian commented 5 years ago

If I understand correctly, firmwares are stored locally. Is it possible via Python script to grab those firmware from Dell website instead of downloading them manually through browser.

texroemer commented 5 years ago

Hi @tyagian

iDRAC firmware updates for devices, you can either store the Dell update package (DUP) locally or on a supported network share. Dell repository update feature as stated in this issue is the easiest way for a user to point their server to a repository which contains the DUPs. You don't have to worry about checking which firmware versions you have installed on your server, iDRAC will handle this internally and only update devices which the firmware is not at the latest version.

If you are looking for a way to download DUPs from Dell support site using python, you can create a custom script using urllib module to do this.

Thanks

tyagian commented 5 years ago

Thank you. Does this script works for Idrac 7/8 also? If yes, what's the firmware version req?

texroemer commented 5 years ago

Currently repository firmware update support using Redfish is only supported with iDRAC 9. If using iDRAC 7/8, as a workaround you can use "racadm update" command for repository updates.

tyagian commented 5 years ago

For racadm update, I have to download DUPs (.exe's) manually. So, nothing exists for idrac 7/8 which can download (install also if possible) it's latest req DUPs by giving idrac IP in argument of script?

texroemer commented 5 years ago

"racadm update" command, you can either point to a single DUP downloaded locally or you can point it to a repository.

Example below is a script i created for a customer which wanted to use repository updates with RACADM. All you need to do is edit the ini file with your iDRAC and repository share information. The ini file is already set up to use downloads.dell.com HTTP site so if you want to use that site, you only need to pass in your iDRAC IP, iDRAC username and password. Once you have edited the ini file, execute the script. Also make sure you have remote RACADM installed first which the script will use.

INI file code, copy these contents into a file named "repository_update_RACADM.ini":

[Parameters] idrac_ip=192.168.0.120 idrac_username=root idrac_password=calvin repo_uri_path=143.166.147.76 protocol_type=HTTP apply_update=TRUE catalog_filename= cifs_username= cifs_password=

Script code, copy these contents into a file named "RepositoryUpdateRACADM.py".

# _author_ = Texas Roemer <Texas_Roemer@Dell.com>
# _version_ = 1.0
#
# Copyright (c) 2019, Dell, Inc.
#
# This software is licensed to you under the GNU General Public License,
# version 2 (GPLv2). There is NO WARRANTY for this software, express or
# implied, including the implied warranties of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. You should have received a copy of GPLv2
# along with this software; if not, see
# http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
#
# Python script using remote RACADM to perform repository firmware updates for devices detected on the server.
#
# You must have remote RACADM installed on the system you are running the python script.
#
# Repository with Dell Update packages and Catalog.xml file already created and is uploaded on the network share you are using. Supported network shares are FTP, TFTP, HTTP, NFS and CIFS
#
# If needed, execute "racadm help updates" to see help text along with examples of the repo update command being used in this script.

import sys, subprocess, time, os, re, ConfigParser

from datetime import datetime

config=ConfigParser.ConfigParser()
config.read("repository_update_RACADM.ini")
idrac_ip=config.get("Parameters","idrac_ip")
idrac_username=config.get("Parameters","idrac_username")
idrac_password=config.get("Parameters","idrac_password")
repo_uri_path=config.get("Parameters","repo_uri_path")
protocol_type=config.get("Parameters","protocol_type")
apply_update=config.get("Parameters","apply_update")

try:
    cifs_username=config.get("Parameters","cifs_username")
except:
    pass
try:
    cifs_password=config.get("Parameters","cifs_password")
except:
    pass
try:
    catalog_filename=config.get("Parameters","catalog_filename")
except:
    pass

default_racadm_command = "racadm -r %s -u %s -p %s --nocertwarn" % (idrac_ip, idrac_username, idrac_password)

def verify_remote_racadm_installed():
    print("\n- WARNING, checking if remote RACADM is installed first before starting the script")
    racadm_command = "%s getversion" % (default_racadm_command)
    zz=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    if "iDRAC Version" in zz:
        print("- PASS, remote RACADM is installed, script will now continue to perform repository updates")
    else:
        print("- FAIL, remote RACADM is not installed. This must be installed to execute the script")
        sys.exit()

def get_job_store_job_ids():
    global job_ids_current
    racadm_command = "%s jobqueue view" % (default_racadm_command)
    zz=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    zz=re.sub("\r","",zz)
    job_id_search = re.findall("JID_.*",zz)
    job_ids_current = []
    for i in job_id_search:
        i=i.replace("]","")
        job_ids_current.append(i)

def perform_repo_update():
    global repo_job_id
    if cifs_username and cifs_password and catalog_filename:
        racadm_command = "%s update -e %s -a %s -t %s -u %s -p %s -f %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type, cifs_username, cifs_password, catalog_filename)
    elif cifs_username and cifs_password:
        racadm_command = "%s update -e %s -a %s -t %s -u %s -p %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type, cifs_username, cifs_password)
    elif catalog_filename:
        racadm_command = "%s update -e %s -a %s -t %s -f %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type, catalog_filename)
    else:
        racadm_command = "%s update -e %s -a %s -t %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type)
    print("\n- WARNING, performing repo update on iDRAC %s using %s URI %s\n" % (idrac_ip, protocol_type, repo_uri_path))
    zz=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    zz=re.sub("\r","",zz)
    if "RAC1118" in zz:
        pass
    else:
        print("- FAIL, repo job ID not created, error is: %s" % zz)
        sys.exit()
    repo_job_id = re.search("JID_.*",zz).group()
    repo_job_id = repo_job_id.replace('" command.','')
    print("- PASS, repo update job %s successfully created. Script will now loop polling the job status every 30 seconds until marked completed" % repo_job_id)

def check_repo_job_status():
    start_time=datetime.now()
    while True:
        racadm_command = "%s jobqueue view -i %s" % (default_racadm_command, repo_job_id)
        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
        z=re.sub("\r","",subprocess_output)
        try:
            job_status=re.search("Status=.+",z).group()
        except:
            print("- FAIL to get job status for job ID %s, trying again" % i)
            sys.exit()

        current_time=str(datetime.now()-start_time)[0:7]
        if current_time >= "0:50:00":
            print("\n- FAIL: Max timeout of 50 minutes reached")
            sys.exit()
        elif job_status == "Status=Completed":
            print("\n- PASS: repo %s job ID completed. Detailed job results -\n" % repo_job_id)
            print(subprocess_output)
            time.sleep(30)
            break

        elif job_status == "Status=Failed":
            print("\n- FAIL: %s job ID failed, error is: %s" % (repo_job_id, subprocess_output))
            sys.exit()
        else:
            print("- WARNING: repo %s job id not marked completed, current status is: %s, current execution time: %s" % (repo_job_id, job_status, current_time))
            time.sleep(30)

def check_repo_update_job_ids():
    print("- WARNING, checking if repo update jobs were created due to firmware version difference detected")
    racadm_command = "%s jobqueue view" % (default_racadm_command)
    subprocess_output=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    z=re.sub("\r","",subprocess_output)
    job_id_search = re.findall("JID_.*",z)
    job_ids_new = []
    for i in job_id_search:
        i=i.replace("]","")
        job_ids_new.append(i)
    job_ids_new.remove(repo_job_id)
    repo_update_job_ids = []
    for i in job_ids_new:
        if i not in job_ids_current:
            repo_update_job_ids.append(i)
        else:
            pass
    if repo_update_job_ids == []:
        print("- WARNING, no update jobs created. Firmware package versions on the repository match the firmware versions on ther server")
        sys.exit()
    else:
        print("- WARNING, repo update job(s) detected. Script will now loop polling each update job ID until marked completed")
        for i in repo_update_job_ids:
            if "RID" in i:
                pass
            else:
                start_time=datetime.now()
                while True:
                    if repo_update_job_ids == []:
                        sys.exit()
                    else:
                        racadm_command = "%s jobqueue view -i %s" % (default_racadm_command, i)
                        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                        z=re.sub("\r","",subprocess_output)
                        try:
                            job_status=re.search("Status=.+",z).group()
                        except:
                            print("- FAIL to get job status for job ID %s, trying again" % i)
                            sys.exit()
                        try:
                            job_name=re.search("Job Name=.+",z).group()
                        except:
                            print("- FAIL to get job name for job ID %s, trying again" % i)
                            sys.exit()
                        current_time=str(datetime.now()-start_time)[0:7]
                        if job_name == "Job Name=Firmware Update: iDRAC":
                            print("- WARNING, iDRAC update job detected. Once the update is complete, iDRAC will reset\n")
                            while True:
                                racadm_command = "%s jobqueue view -i %s" % (default_racadm_command, i)
                                subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                                subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                                z=re.sub("\r","",subprocess_output)
                                try:
                                    job_status=re.search("Status=.+",z).group()
                                except:
                                    print("- WARNING, either slow network connection detected or iDRAC connection lost due to reset of the iDRAC after updated completed, script complete.")
                                    print("\n- Once the iDRAC is back up, you can manually check the job status to verify iDRAC update completed successfully using \"racadm jobqueue view -i %s\" command" % i)
                                    sys.exit()
                                try:
                                    job_name=re.search("Job Name=.+",z).group()
                                except:
                                    print("- WARNING, either slow network connection detected or iDRAC connection lost due to reset of the iDRAC after updated completed, script complete.")
                                    print("\n- Once the iDRAC is back up, you can manually check the job status to verify iDRAC update completed successfully using \"racadm jobqueue view -i %s\" command" % i)
                                    sys.exit()
                                current_time=str(datetime.now()-start_time)[0:7]   
                                if current_time >= "0:50:00":
                                    print("- FAIL: Max timeout of 50 minutes reached")
                                    sys.exit()
                                elif job_status == "Status=Completed":
                                    print("\n- PASS: update %s job ID completed. Detailed job results -\n" % i)
                                    print(subprocess_output)
                                    sys.exit()
                                elif job_status == "Status=Failed":
                                    print("\n- FAIL: %s jid failed, detailed job results -\n" % i)
                                    print(subprocess_output)
                                    sys.exit()
                                else:
                                    print("- WARNING: update %s job id not marked completed, current status is: %s, current execution time: %s" % (i, job_status, current_time))
                                    continue
                        else:   
                            if current_time >= "0:50:00":
                                print("- FAIL: Max timeout of 50 minutes reached")
                                sys.exit()
                            elif job_status == "Status=Completed":
                                print("\n- PASS: update %s job ID completed. Detailed job results -\n" % i)
                                print(subprocess_output)
                                repo_update_job_ids.remove(i)
                                break
                            elif job_status == "Status=Failed":
                                print("\n- FAIL: %s jid failed, detailed job results -\n" % i)
                                print(subprocess_output)
                                repo_update_job_ids.remove(i)
                                break
                            else:
                                print("- WARNING: update %s job id not marked completed, current status is: %s, current execution time: %s" % (i, job_status, current_time))
                                continue

if __name__ == "__main__":
    verify_remote_racadm_installed()
    get_job_store_job_ids()
    perform_repo_update()
    check_repo_job_status()
    check_repo_update_job_ids()
    print("\n- Execution of Repository Update script complete")