search

bseltz-cohesity / scripts

Cohesity REST API examples in PowerShell and Python
Apache License 2.0
102 stars 41 forks source link

instantVolumeMount.py KeyError with runId #147

Closed dragongeek closed 7 months ago

dragongeek commented 7 months ago

Apologies for the two part question.

  1. When I try to run instantVolumeMount.py with a specific runID I get the error listed below my questions. I updated our Cohesity Appliance to version 7.1.1, and downloaded all the latest Python code after upgrading. This error started after the upgrades.

# /root/cohesity/bin/instantVolumeMount.py -v cohesitybkup -u apiuser -d local -s titan -t titan -r 5779489 Connected! Traceback (most recent call last): File "/root/cohesity/bin/instantVolumeMount.py", line 143, in snapshots['snapshots'] = [s for s in snapshots['snapshots'] if s['runId'] == runid] KeyError: 'runId'

  1. I also noticed that I cannot use the Protection Job Run ID, which was a pair of colon separated numbers, as you changed the variable for the runId to an Int from a Str. I was capturing the output from the protection job run and parsing it for my scripts to kick off the Instant Volume Mount after the backup finished. I would like to know if there is a different way to use the Job Run Id, or we have to now query the backup IDs with backedUpFileList.py to get the run ID going forward?

Output example from ./backupNow.py run showing the information I was parsing to start the instantVolumeMounts:

New Job Run ID: 1350099:1711778417377068 Job finished with status: Succeeded

Thank you, Michael

bseltz-cohesity commented 7 months ago

Hi @dragongeek OK I have fixed the script. It will now work whether you provide a v1 runId (as listed when using the -sh (--showversions) option, or the v2 runId as shown by backupNow.

dragongeek commented 7 months ago

It launches, but it still seems to be doing something odd, because while I can run an IVM from the Web UI, if I launch it from the instantVolumeMount.py (with or without passing the ID) it doesn't complete the mount, but repeats as below until I cancel it (even after running for several hours).

instantVolumeMount.py -v cohesitybkup -u apiuser -d local -s titan -t titan -e kVMware --useexistingagent -r 5783189

LOG OUTPUT: Job is being validated Scheduling task of type: kValidateObjects, task_id: 1 Scheduling task of type: kValidateObjects, task_id: 2 Scheduling task of type: kValidateObjects, task_id: 3 Scheduling task of type: kValidateObjects, task_id: 4 Scheduling task of type: kValidateObjects, task_id: 5 Scheduling task of type: kValidateObjects, task_id: 6 Scheduling task of type: kValidateObjects, task_id: 7 Scheduling task of type: kValidateObjects, task_id: 8 Scheduling task of type: kValidateObjects, task_id: 9 Scheduling task of type: kValidateObjects, task_id: 10 Scheduling task of type: kValidateObjects, task_id: 11 Scheduling task of type: kValidateObjects, task_id: 12 Scheduling task of type: kValidateObjects, task_id: 13 Scheduling task of type: kValidateObjects, task_id: 14

bseltz-cohesity commented 7 months ago

Hi, I am not able to reproduce your issue. I've added a -debug, --debug command line option. The updated script is here: https://github.com/bseltz-cohesity/scripts/tree/master/python/instantVolumeMount Please add --debug to your command and it will display the JSON payload, please post the JSON so I can review. Thanks.

dragongeek commented 7 months ago

I tried again against another server from the same protection job after disabling the firewall and SELinux, but I get the same result whether I give the v1 runID, v2, or no runID at all. The command line shows it just sitting after it echos the tearDown ID:

[root@radar bin]# ./instantVolumeMount.py -v liscohobkup -u apiuser -d local -s kiska -t kiska --useexistingagent --wait --debug -r 5782715 Connected! Performing instant volume mount... { "name": "Recover_kiska_to_kiska_2024-04-08_10:03:47", "snapshotEnvironment": "kVMware", "vmwareParams": { "mountVolumeParams": { "targetEnvironment": "kVMware", "vmwareTargetParams": { "mountToOriginalTarget": true, "newTargetConfig": null, "originalTargetConfig": { "bringDisksOnline": true, "targetVmCredentials": null, "useExistingAgent": true }, "readOnlyMount": false, "volumeNames": null } }, "objects": [ { "snapshotId": "eyJhX2NsdXN0ZXJJZCI6MjgzMjM4ODQ5NzQ4MTQ2OCwiYl9jbHVzdGVySW5jYXJuYXRpb25JZCI6MTU0OTkxNDcxOTcyOSwiY19qb2JJZCI6MTM1MDA5OSwiZV9qb2JJbnN0YW5jZUlkIjo1NzgyNzE1LCJmX3J1blN0YXJ0VGltZVVzZWNzIjoxNzEyMTAzNTg3MzY0NjcyLCJnX29iamVjdElkIjozNjc3LCJoX3ZhdWx0SWQiOjU4NTkzNDQ0NzZ9" } ], "recoveryAction": "InstantVolumeMount" } } Task ID for tearDown is: 5806187

It will sit there until it gets cancelled, but the Web UI Log shows:

Recovery Activity for Recover_kiska_to_kiska_2024-04-08_10:03:47 April 8, 2024 (Today) 10:03 AM

Retrieving objects from archive
Job is being validated
Scheduling task of type: kValidateObjects, task_id: 1
Scheduling task of type: kValidateObjects, task_id: 2
Scheduling task of type: kValidateObjects, task_id: 3
Scheduling task of type: kValidateObjects, task_id: 4
Scheduling task of type: kValidateObjects, task_id: 5
Scheduling task of type: kValidateObjects, task_id: 6
Scheduling task of type: kValidateObjects, task_id: 7
Scheduling task of type: kValidateObjects, task_id: 8
Scheduling task of type: kValidateObjects, task_id: 9
Scheduling task of type: kValidateObjects, task_id: 10
Scheduling task of type: kValidateObjects, task_id: 11
Scheduling task of type: kValidateObjects, task_id: 12
Scheduling task of type: kValidateObjects, task_id: 13
Scheduling task of type: kValidateObjects, task_id: 14
Scheduling task of type: kValidateObjects, task_id: 15
Scheduling task of type: kValidateObjects, task_id: 16
Scheduling task of type: kValidateObjects, task_id: 17
Scheduling task of type: kValidateObjects, task_id: 18
Scheduling task of type: kValidateObjects, task_id: 19
Scheduling task of type: kValidateObjects, task_id: 20
Scheduling task of type: kValidateObjects, task_id: 21
Scheduling task of type: kValidateObjects, task_id: 22
Scheduling task of type: kValidateObjects, task_id: 23
Scheduling task of type: kValidateObjects, task_id: 24
Scheduling task of type: kValidateObjects, task_id: 25
Scheduling task of type: kValidateObjects, task_id: 26
Scheduling task of type: kValidateObjects, task_id: 27
Scheduling task of type: kValidateObjects, task_id: 28
Scheduling task of type: kValidateObjects, task_id: 29
Scheduling task of type: kValidateObjects, task_id: 30
Scheduling task of type: kValidateObjects, task_id: 31
Scheduling task of type: kValidateObjects, task_id: 32
Scheduling task of type: kValidateObjects, task_id: 33
Scheduling task of type: kValidateObjects, task_id: 34
Scheduling task of type: kValidateObjects, task_id: 35
Scheduling task of type: kValidateObjects, task_id: 36
Scheduling task of type: kValidateObjects, task_id: 37
Scheduling task of type: kValidateObjects, task_id: 38
Scheduling task of type: kValidateObjects, task_id: 39
Scheduling task of type: kValidateObjects, task_id: 40
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 5, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 37, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 1, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 9, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 21, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 25, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 13, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 29, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 33, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 17, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 2, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 18, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 14, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 22, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 34, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 6, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 32, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 38, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 10, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 20, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 28, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 36, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 30, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 26, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 4, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 8, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 16, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 24, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 40, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 12, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 11, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 7, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 31, task type: kValidateObjects
Scheduling task of type: kValidateObjects, task_id: 41
Scheduling task of type: kValidateObjects, task_id: 42
Scheduling task of type: kValidateObjects, task_id: 43
Scheduling task of type: kValidateObjects, task_id: 44
Scheduling task of type: kValidateObjects, task_id: 45
Scheduling task of type: kValidateObjects, task_id: 46
Scheduling task of type: kValidateObjects, task_id: 47
Scheduling task of type: kValidateObjects, task_id: 48
Scheduling task of type: kValidateObjects, task_id: 49
Scheduling task of type: kValidateObjects, task_id: 50
Scheduling task of type: kValidateObjects, task_id: 51
Scheduling task of type: kValidateObjects, task_id: 52
Scheduling task of type: kValidateObjects, task_id: 53
Scheduling task of type: kValidateObjects, task_id: 54
Scheduling task of type: kValidateObjects, task_id: 55
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 39, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 35, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 3, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 19, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 27, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 15, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 23, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 55, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 53, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 41, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 47, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 50, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 44, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 43, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 49, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 46, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 52, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 45, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 42, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 54, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 51, task type: kValidateObjects
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 48, task type: kValidateObjects
Job preprocessing is in progress
Created kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 57
Created kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 58
Created kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 59
Created kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 60
Scheduling task of type: kPreprocessJob, task_id: 56
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 56, task type: kPreprocessJob
Job is currently running
Scheduling task of type: kListVaultFSFiles, task_id: 57
Scheduling task of type: kListVaultFSFiles, task_id: 58
Scheduling task of type: kListVaultFSFiles, task_id: 59
Scheduling task of type: kListVaultFSFiles, task_id: 60
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 59, task type: kListVaultFSFiles
Completed kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 59
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 58, task type: kListVaultFSFiles
Completed kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 58
Setting task complete - job_id: 2832388497481468:1549914719729:83382492389, job_type: kRestore, task_id: 60, task type: kListVaultFSFiles
Completed kListVaultFSFiles task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 60
Created kRestoreDirectories task for icebox job: 2832388497481468:1549914719729:83382492389, task id: 61
Scheduling task of type: kRestoreDirectories, task_id: 61

[ ... this continues until Cancelled ]

bseltz-cohesity commented 7 months ago

Aha! What I didn't see was: "Retrieving objects from archive" that's what was taking so long. I have now updated the script to pull only from local snapshots unless you specify -x (--usearchive). Updated script is here: https://github.com/bseltz-cohesity/scripts/tree/master/python/instantVolumeMount

dragongeek commented 7 months ago

Fantastic! Thanks Brian! That did the trick alright!