dell / iDRAC-Redfish-Scripting

Python and PowerShell scripting for Dell EMC PowerEdge iDRAC REST API with DMTF Redfish
GNU General Public License v2.0
606 stars 279 forks source link

Redfish script to upgrade BIOS, Controller and Idrac #39

Closed tyagian closed 5 years ago

tyagian commented 6 years ago

I am using Idrac 7, 8 and 9 on R730xd and need to upgrade BIOS, controllers, and Idrac in bulk of servers. It would be great if we have Redfish or non-redfish based Python script to upgrade them. Or just script to upload iso/ exe which we usually do to upgrade them.

texroemer commented 6 years ago

Hi @tyagian,

Redfish update support was added to 2.60 iDRAC 7/8 release so if your servers are at an older iDRAC version, you won't be able to use Redfish to perform updates. So the easiest way to get your servers up to date to latest firmware versions is use repository updates with remote RACADM. The command you would use is "racadm update" which you would point to either a custom repository you create or to the Dell repository which is recommended since the repository is already set up and has the latest firmware packages.

I created a Python script which uses remote RACADM update command and points to the Dell repository. The script calls a config.ini which i already configured to use the Dell repository. All you need to modify in the config.ini is idrac_ip, idrac_username and idrac_password parameters. Then from command line, execute the Python script which will call the config.file to perform updates using a repository. Before executing the Python script, you must have remote RACADM installed (you can get this from Dell support site).

If you don't want to use Dell repository and want to create your own custom repository, you can use Dell Repository Manager utility. This is also available on Dell support site to download. Also if you do decide to use a custom repository, you will need to modify the other parameters in the config.ini file. From command line, run "racadm help update" which will return help text. In the help text it will show you supported protocols along with other parameter values.

"repository_update_RACADM.ini" file (make sure to name the ini file with this name because the script will look for this exact file name):

[Parameters]
idrac_ip=192.168.0.120
idrac_username=root
idrac_password=calvin
repo_uri_path=143.166.135.76
protocol_type=HTTP
apply_update=TRUE
catalog_filename=Catalog.xml.gz
cifs_username=
cifs_password=

Python script "RepositoryUpdateRACADM.py":

import sys, subprocess, time, os, re, ConfigParser

from datetime import datetime

config=ConfigParser.ConfigParser()
config.read("repository_update_RACADM.ini")
idrac_ip=config.get("Parameters","idrac_ip")
idrac_username=config.get("Parameters","idrac_username")
idrac_password=config.get("Parameters","idrac_password")
repo_uri_path=config.get("Parameters","repo_uri_path")
protocol_type=config.get("Parameters","protocol_type")
apply_update=config.get("Parameters","apply_update")

try:
    cifs_username=config.get("Parameters","cifs_username")
except:
    pass
try:
    cifs_password=config.get("Parameters","cifs_password")
except:
    pass
try:
    catalog_filename=config.get("Parameters","catalog_filename")
except:
    pass

default_racadm_command = "racadm -r %s -u %s -p %s --nocertwarn" % (idrac_ip, idrac_username, idrac_password)

def verify_remote_racadm_installed():
    print("\n- WARNING, checking if remote RACADM is installed first before starting the script")
    racadm_command = "%s getversion" % (default_racadm_command)
    zz=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    if "iDRAC Version" in zz:
        print("- PASS, remote RACADM is installed, script will now continue to perform repository updates")
    else:
        print("- FAIL, remote RACADM is not installed. This must be installed to execute the script")
        sys.exit()

def get_job_store_job_ids():
    global job_ids_current
    racadm_command = "%s jobqueue view" % (default_racadm_command)
    zz=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    zz=re.sub("\r","",zz)
    job_id_search = re.findall("JID_.*",zz)
    job_ids_current = []
    for i in job_id_search:
        i=i.replace("]","")
        job_ids_current.append(i)

def perform_repo_update():
    global repo_job_id
    if cifs_username and cifs_password and catalog_filename:
        racadm_command = "%s update -e %s -a %s -t %s -u %s -p %s -f %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type, cifs_username, cifs_password, catalog_filename)
    elif cifs_username and cifs_password:
        racadm_command = "%s update -e %s -a %s -t %s -u %s -p %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type, cifs_username, cifs_password)
    elif catalog_filename:
        racadm_command = "%s update -e %s -a %s -t %s -f %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type, catalog_filename)
    else:
        racadm_command = "%s update -e %s -a %s -t %s" % (default_racadm_command, repo_uri_path, apply_update, protocol_type)
    print("\n- WARNING, performing repo update on iDRAC %s using %s URI %s\n" % (idrac_ip, protocol_type, repo_uri_path))
    zz=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    zz=re.sub("\r","",zz)
    if "RAC1118" in zz:
        pass
    else:
        print("- FAIL, repo job ID not created, error is: %s" % zz)
        sys.exit()
    repo_job_id = re.search("JID_.*",zz).group()
    repo_job_id = repo_job_id.replace('" command.','')
    print("- PASS, repo update job %s successfully created. Script will now loop polling the job status every 30 seconds until marked completed" % repo_job_id)

def check_repo_job_status():
    start_time=datetime.now()
    while True:
        racadm_command = "%s jobqueue view -i %s" % (default_racadm_command, repo_job_id)
        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
        z=re.sub("\r","",subprocess_output)
        try:
            job_status=re.search("Status=.+",z).group()
        except:
            print("- FAIL to get job status for job ID %s, trying again" % i)
            sys.exit()

        current_time=str(datetime.now()-start_time)[0:7]
        if current_time >= "0:50:00":
            print("\n- FAIL: Max timeout of 50 minutes reached")
            sys.exit()
        elif job_status == "Status=Completed":
            print("\n- PASS: repo %s job ID completed. Detailed job results -\n" % repo_job_id)
            print(subprocess_output)
            time.sleep(30)
            break

        elif job_status == "Status=Failed":
            print("\n- FAIL: %s job ID failed, error is: %s" % (repo_job_id, subprocess_output))
            sys.exit()
        else:
            print("- WARNING: repo %s job id not marked completed, current status is: %s, current execution time: %s" % (repo_job_id, job_status, current_time))
            time.sleep(30)

def check_repo_update_job_ids():
    print("- WARNING, checking if repo update jobs were created due to firmware version difference detected")
    racadm_command = "%s jobqueue view" % (default_racadm_command)
    subprocess_output=subprocess.Popen(racadm_command,stdout=subprocess.PIPE,shell=True).communicate()[0]
    z=re.sub("\r","",subprocess_output)
    job_id_search = re.findall("JID_.*",z)
    job_ids_new = []
    for i in job_id_search:
        i=i.replace("]","")
        job_ids_new.append(i)
    job_ids_new.remove(repo_job_id)
    repo_update_job_ids = []
    for i in job_ids_new:
        if i not in job_ids_current:
            repo_update_job_ids.append(i)
        else:
            pass
    if repo_update_job_ids == []:
        print("- WARNING, no update jobs created. Firmware package versions on the repository match the firmware versions on ther server")
        sys.exit()
    else:
        print("- WARNING, repo update job(s) detected. Script will now loop polling each update job ID until marked completed")
        for i in repo_update_job_ids:
            if "RID" in i:
                pass
            else:
                start_time=datetime.now()
                while True:
                    if repo_update_job_ids == []:
                        sys.exit()
                    else:
                        racadm_command = "%s jobqueue view -i %s" % (default_racadm_command, i)
                        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                        subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                        z=re.sub("\r","",subprocess_output)
                        try:
                            job_status=re.search("Status=.+",z).group()
                        except:
                            print("- FAIL to get job status for job ID %s, trying again" % i)
                            sys.exit()
                        try:
                            job_name=re.search("Job Name=.+",z).group()
                        except:
                            print("- FAIL to get job name for job ID %s, trying again" % i)
                            sys.exit()
                        current_time=str(datetime.now()-start_time)[0:7]
                        if job_name == "Job Name=Firmware Update: iDRAC":
                            print("- WARNING, iDRAC update job detected. Once the update is complete, iDRAC will reset\n")
                            while True:
                                racadm_command = "%s jobqueue view -i %s" % (default_racadm_command, i)
                                subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                                subprocess_output=subprocess.Popen(racadm_command, stdout=subprocess.PIPE, shell=True).communicate()[0]
                                z=re.sub("\r","",subprocess_output)
                                try:
                                    job_status=re.search("Status=.+",z).group()
                                except:
                                    print("- WARNING, either slow network connection detected or iDRAC connection lost due to reset of the iDRAC after updated completed, script complete.")
                                    print("\n- Once the iDRAC is back up, you can manually check the job status to verify iDRAC update completed successfully using \"racadm jobqueue view -i %s\" command" % i)
                                    sys.exit()
                                try:
                                    job_name=re.search("Job Name=.+",z).group()
                                except:
                                    print("- WARNING, either slow network connection detected or iDRAC connection lost due to reset of the iDRAC after updated completed, script complete.")
                                    print("\n- Once the iDRAC is back up, you can manually check the job status to verify iDRAC update completed successfully using \"racadm jobqueue view -i %s\" command" % i)
                                    sys.exit()
                                current_time=str(datetime.now()-start_time)[0:7]   
                                if current_time >= "0:50:00":
                                    print("- FAIL: Max timeout of 50 minutes reached")
                                    sys.exit()
                                elif job_status == "Status=Completed":
                                    print("\n- PASS: update %s job ID completed. Detailed job results -\n" % i)
                                    print(subprocess_output)
                                    sys.exit()
                                elif job_status == "Status=Failed":
                                    print("\n- FAIL: %s jid failed, detailed job results -\n" % i)
                                    print(subprocess_output)
                                    sys.exit()
                                else:
                                    print("- WARNING: update %s job id not marked completed, current status is: %s, current execution time: %s" % (i, job_status, current_time))
                                    continue
                        else:   
                            if current_time >= "0:50:00":
                                print("- FAIL: Max timeout of 50 minutes reached")
                                sys.exit()
                            elif job_status == "Status=Completed":
                                print("\n- PASS: update %s job ID completed. Detailed job results -\n" % i)
                                print(subprocess_output)
                                repo_update_job_ids.remove(i)
                                break
                            elif job_status == "Status=Failed":
                                print("\n- FAIL: %s jid failed, detailed job results -\n" % i)
                                print(subprocess_output)
                                repo_update_job_ids.remove(i)
                                break
                            else:
                                print("- WARNING: update %s job id not marked completed, current status is: %s, current execution time: %s" % (i, job_status, current_time))
                                continue

if __name__ == "__main__":
    verify_remote_racadm_installed()
    get_job_store_job_ids()
    perform_repo_update()
    check_repo_job_status()
    check_repo_update_job_ids()
    print("\n- Execution of Repository Update script complete") 
tyagian commented 6 years ago

Great. Thanks a lot. Did you just write it now?
I found 2 relevant scripts (DeviceFirmwareDellUpdateServiceREDFISH and DeviceFirmwareSimpleUpdateREDFISH.py) but this one is new and not available in the repository yet.

texroemer commented 6 years ago

New RACADM script i just created since you will have servers at an iDRAC version which doesn't have Redfish update support. Once you update iDRAC 7/8 to 2.60, you can then use the Redfish scripts you mentioned. If you have 14G servers with iDRAC 9, you can already use the Redfish scripts but the RACADM script i just created will also work.

tyagian commented 6 years ago

Appreciate your efforts.

texroemer commented 6 years ago

No problem, glad to help!

tyagian commented 6 years ago

Do you also know any way to upgrade BIOS or IDRAC (great if both) also using a script? This only upgrade firmware which is great but need to upgrade BIOS, Idrac and maybe backplane also for few. There are Idrac 8 and 9 both in different environments.

texroemer commented 6 years ago

Once at the latest iDRAC versions i mentioned, you can use script "DeviceFirmwareSimpleUpdateREDFISH.py" to perform firmware updates for devices.

Example below is using this script to update the BIOS on R730 from 2.7.1 to 2.8.0.

C:\Python27>DeviceFirmwareSimpleUpdateREDFISH.py -ip 192.168.0.120 -u root -p calvin -g y

Device Name: Power Supply.Slot.1, Firmware Version: 00.10.37, Updatable: True Device Name: BP13G+ 0:1, Firmware Version: 2.25, Updatable: True Device Name: PERC H730 Mini, Firmware Version: 25.5.5.0005, Updatable: True Device Name: OS COLLECTOR 1.1, OSC_1.1, A00, Firmware Version: OSC_1.1, Updatable: True Device Name: Disk 5 in Backplane 1 of Integrated RAID Controller 1, Firmware Version: VT31, Updatable: True Device Name: BIOS, Firmware Version: 2.7.1, Updatable: True Device Name: Dell OS Driver Pack, 17.08.10, A00, Firmware Version: 17.08.10, Updatable: True Device Name: Integrated Dell Remote Access Controller, Firmware Version: 2.60.60.60, Updatable: True Device Name: Diagnostics, Firmware Version: 0, Updatable: True Device Name: QLogic 577xx/578xx 10 Gb Ethernet BCM57810 - 00:0A:F7:5D:7A:72, Firmware Version: 14.04.18, Updatable: True Device Name: Broadcom Gigabit Ethernet BCM5720 - 44:A8:42:46:9C:2B, Firmware Version: 7.10.64, Updatable: True Device Name: System CPLD, Firmware Version: 1.0.1, Updatable: True Device Name: Lifecycle Controller, Firmware Version: 2.60.60.60, Updatable: False

C:\Python27>DeviceFirmwareSimpleUpdateREDFISH.py -ip 192.168.0.120 -u root -p calvin -l C:\Python27\master_scripts_modified -f BIOS_2JFRF_WN64_2.8.0.EXE

@odata.type: #DellJob.v1_0_0.DellJob JobState: Completed Description: Job Instance TargetSettingsURI: None MessageArgs: [] CompletionTime: 2018-11-09T20:41:37 PercentComplete: 100 StartTime: TIME_NOW MessageId: PR19 Message: Job completed successfully. EndTime: TIME_NA Id: JID_418172050541 JobType: FirmwareUpdate Name: update:DCIM:INSTALLED#741__BIOS.Setup.1-1

C:\Python27>DeviceFirmwareSimpleUpdateREDFISH.py -ip 192.168.0.120 -u root -p calvin -g y

Device Name: Power Supply.Slot.1, Firmware Version: 00.10.37, Updatable: True Device Name: BP13G+ 0:1, Firmware Version: 2.25, Updatable: True Device Name: PERC H730 Mini, Firmware Version: 25.5.5.0005, Updatable: True Device Name: OS COLLECTOR 1.1, OSC_1.1, A00, Firmware Version: OSC_1.1, Updatable: True Device Name: Disk 5 in Backplane 1 of Integrated RAID Controller 1, Firmware Version: VT31, Updatable: True Device Name: BIOS, Firmware Version: 2.8.0, Updatable: True Device Name: Dell OS Driver Pack, 17.08.10, A00, Firmware Version: 17.08.10, Updatable: True Device Name: Integrated Dell Remote Access Controller, Firmware Version: 2.60.60.60, Updatable: True Device Name: Diagnostics, Firmware Version: 0, Updatable: True Device Name: QLogic 577xx/578xx 10 Gb Ethernet BCM57810 - 00:0A:F7:5D:7A:72, Firmware Version: 14.04.18, Updatable: True Device Name: Broadcom Gigabit Ethernet BCM5720 - 44:A8:42:46:9C:2B, Firmware Version: 7.10.64, Updatable: True Device Name: System CPLD, Firmware Version: 1.0.1, Updatable: True Device Name: Lifecycle Controller, Firmware Version: 2.60.60.60, Updatable: False

C:\Python27>

tyagian commented 6 years ago

oh! it updates all of them? It seems like yum upgrade

texroemer commented 6 years ago

The RACADM repo script i gave you will update any device which detects a firmware version difference on the server against the repository it's using.

DeviceFirmwareSimpleUpdateREDFISH.py script will just update one device at a time but it can be modified to update multiple devices as needed. The example i posted using this script is only updating the BIOS firmware. Before and after running the script, i pulled the firmware of all devices but just wanted to highlight in bold text the BIOS version did change after the update.

tyagian commented 6 years ago

so I am testing RepositoryUpdateRACADM.py script but how did you find catalog_filename for .ini file? Also, Can I use the current directory as repo_uri_path?

texroemer commented 6 years ago

For the repo catalog filename, iDRAC code will look for one of the 2 default catalog filenames which are "Catalog.xml" or "Catalog.xml.gz". If the catalog file is one of these names, you can either pass it in or leave this parameter field empty. If you decide to change the name of the catalog file to a custom name, then its mandatory you pass in this name.

For the Dell repository, its using "Catalog.xml.gz" which is a default catalog file name. When the repository is generated using Dell Repository Manager utility, it will generate a "Catalog.xml" file for you when you want to save the repository. Due to how many platforms the Dell repository is using, we zipped the catalog file.

Repository update only supports network shares so local directories are not supported. If you want to use your current directory, convert it to one of the supported network shares and make sure it contains the firmware Dell Update packages (Windows .exe only supported) along with the Catalog.xml file. You will need to use Dell Repository Manager to generate the Catalog.xml file.

aviv12825 commented 5 years ago

Hi, Maybe this have been answered before... but Im trying to use the DeviceFirmwareSimpleUpdateREDFISH.py script and getting the following error: C:\Python36>DeviceFirmwareSimpleUpdateREDFISH.py -ip 10.0.0.60 -u root -p calvin -l C:\Python27\master_scripts_modified -f iDRAC-with-Lifecycle-Controller_Firmw are_K877V_WN64_3.21.23.22_A00.EXE

Can anyone exaplin whats wrong here ? **using iDrac9 with 3.17.17.17 fw Thanks

texroemer commented 5 years ago

Hi @aviv12825

3.17 did not have support for HttpPushUri with SimpleUpdate. You will need to use either iDRAC GUI or RACADM to update the iDRAC FW to 3.21. Then try the script again which will now pass.

Also we just posted 3.30 version on 3/4 which is the latest iDRAC version. This firmware version has added lots of new features to Redfish (OEM and DTMF). We are currently working on the docs and scripts for these new features and will have them posted in the next week.

Thanks

aviv12825 commented 5 years ago

Hi @texroemer - thank you for the replay! :) What about idrac8 2.60.60.60 - is it support SimpleUpdate ? I assume the is asnwer is NO - any workaround for this ? Thank you.

texroemer commented 5 years ago

@aviv12825

2.60 iDRAC does support SimpleUpdate. Run a GET on URI "redfish/v1/UpdateService", you will see SimpleUpdate action along with HttpPushUri listed.

Thanks

tyagian commented 5 years ago

I came to know about Dell repo manager now and understood what you were talking about. In the first script at the top, RepositoryUpdateRACADM.py We use the source file to get all user/pass info. If I've repo manager installed on the same system from which I am running this script, it's a vbox ubuntu os where I've setup dell repo manager. What does cifs_username= and cifs_password= used for? Is it for nfs usr/pass? I don't need it if I am running script and dell repo manager from same system?

"repository_update_RACADM.ini"

[Parameters]
idrac_ip=192.168.0.120
idrac_username=root
idrac_password=calvin
repo_uri_path=143.166.135.76
protocol_type=HTTP
apply_update=TRUE
catalog_filename=Catalog.xml.gz
cifs_username=
cifs_password=

When I am trying to run this script, I get error:

$python RepositoryUpdateRACADM.py

texroemer commented 5 years ago

Hi @tyagian

You need to have remote RACADM installed on the system you are running the script from. SSH to the iDRAC and running RACADM command is local RACADM.

Remote RACADM is available to download from Dell Support Site.

Example link for R740xd:

https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=g3ndf&oscode=rh60&productcode=poweredge-r740xd

Thanks

tyagian commented 5 years ago

So when I am using RepositoryUpdateRACADM.py, it expects dell repo manager on NFS and idrac try to reach there. What if I am installing dell repo manager and python script from the same system? It shouldn't req cfs IP, user password? Because when I am using DeviceFirmwareSimpleUpdateREDFISH.py, it doesn't ask req idrac to ask for NFS and other credentials.

Is it possible for a script like DeviceFirmwareSimpleUpdateREDFISH.py to work with dell repo manager? My idrac job failing to reach dell repo manager but it works when I have firmware on the same server and using DeviceFirmwareSimpleUpdateREDFISH.py

texroemer commented 5 years ago

Ubuntu is currently an OS we official support for RACADM but I did a Google search and found multiple workaround articles about installing RACADM on Ubuntu. Can you try this out and see if it works?

https://www.gocit.vn/bai-viet/install-racadm-on-ubuntu-debian-for-dell-idrac/

For the RACADM repo update script, it supports NFS, CIFS, HTTP, TFTP or FTP for network protocol where the repository is located. The INI file example i sent you uses the Dell HTTP repository which has the latest DUPs for your server. Were you not able to access this repository with the script or are you wanting to create your own custom repository?