Commvault / cvpysdk

Developer SDK - Python
https://commvault.github.io/cvpysdk/
Apache License 2.0
54 stars 41 forks source link

Getting a list of jobs for a storage policy copy #145

Closed asshutossh closed 2 years ago

asshutossh commented 2 years ago

from cvpysdk.commcell import Commcell commcell = Commcell("...") from cvpysdk.policies.storage_policies import StoragePolicy sp = StoragePolicy(commcell,"bk-c","129") secondaryCopies = sp.get_secondary_copies()

from cvpysdk.policies.storage_policies import StoragePolicyCopy spc = StoragePolicyCopy(commcell,"bk-c","copy2")

I want to use this function but it requires the job_id. do_not_copy_jobs: (job_id: Any) -> None Marks job(s) as Do Not Copy to a secondary copy

Args: job_id (int or str or list): Job Id(s) that needs to be marked

There is no function to get all jobs per copy. from cvpysdk.job import JobController gives a dictionary of all jobs but it doesnt have any reference to the storage policy. I must select the clients associated with the storage policy to get a list of all jobs per client ?

What I need to do is the following: storage policy hast copy1, copy2 and copy3. I need to only copy certain jobs from copy2 to copy3. So: Select a storage policy -> copy -> Get list of jobs^ids with status "Available Retained" Select Only mark the above jobs for copying by using pick_for_copy() for above jobs run auxcopy for the destination copy

spakhare commented 2 years ago

Hi @asshutossh,

We are checking this. We will get back once we have any findings.

Thanks,

asshutossh commented 2 years ago

Hi @spakhare This could be worked around by getting the results by running a report. So I was trying to run a storage policy copy jobs report by calling the get function with the dataset of this report. But unfortunately its giving me a 404.

I got the dataset ID from the report REST API button as described on https://documentation.commvault.com/v11/essential/115046_rest_api_get_data_set.html:

https:///commandcenter/api/cr/reportsplusengine/datasets/695DADC9-BD89-4B86-B294-AF04D175FD06:1151d30e-fd18-4536-b115-73df02b973fc/data/?offset=0&fields=%5Bsys_rowid%5D%20AS%20%5Bsys_rowid%5D%2C%5BData%20Source%5D%20AS%20%5BDataSource%5D%2C%5BadminConsoleURL%5D%20AS%20%5BadminConsoleURL%5D%2C%5BParentJobID%5D%20AS%20%5BParentJobID1%5D%2C%5BCommcellID%5D%20AS%20%5BCommcellID%5D%2C%5BMediaAgent%5D%20AS%20%5BMediaAgent%5D%2C%5BStorage%20Policy%5D%20AS%20%5BStoragePolicy%5D%2C%5BStorage%20Policy%20Copy%5D%20AS%20%5BStoragePolicyCopy%5D%2C%5BJobID%5D%20AS%20%5BJobID%5D%2C%5BRetention%5D%20AS%20%5BRetention%5D%2C%5BSPCAgedStatus%5D%20AS%20%5BSPCAgedStatus%5D%2C%5BJobStatus%5D%20AS%20%5BJobStatus%5D%2C%5BAuxCopyStatus%5D%20AS%20%5BAuxCopyStatus%5D%2C%5BPhases%5D%20AS%20%5BPhases%5D%2C%5BClientId%5D%20AS%20%5BClientId%5D%2C%5BClient%5D%20AS%20%5BClient%5D%2C%5BHostName%5D%20AS%20%5BHostName%5D%2C%5BAgentType%5D%20AS%20%5BAgentType%5D%2C%5BInstance%5D%20AS%20%5BInstance%5D%2C%5BBackupSet%5D%20AS%20%5BBackupSet%5D%2C%5BSubClient%5D%20AS%20%5BSubClient%5D%2C%5BFilters%5D%20AS%20%5BFilters%5D%2C%5BBackupType%5D%20AS%20%5BBackupType%5D%2C%5BCycle_Seq%5D%20AS%20%5BCycle_Seq%5D%2C%5BStartTime%5D%20AS%20%5BStartTime%5D%2C%5BEndTime%5D%20AS%20%5BEndTime%5D%2C%5BRunLength%20(Sec)%5D%20AS%20%5BRunLengthSec%5D%2C%5BApplication%20Size%5D%20AS%20%5BApplicationSize%5D%2C%5BBackup%20Size%20(GB)%5D%20AS%20%5BBackupSizeGB%5D%2C%5BTransferred%20Size%5D%20AS%20%5BTransferredSize%5D%2C%5BData%20Written%5D%20AS%20%5BDataWritten%5D%2C%5BBarcode%5D%20AS%20%5BBarcode%5D%2C%5BMedia%5D%20AS%20%5BMedia%5D%2C%5BSize%20on%20Media%5D%20AS%20%5BSizeonMedia%5D%2C%5BDataStatus%5D%20AS%20%5BDataStatus%5D%2C%5BDedup%20Saving(%25)%5D%20AS%20%5BDedupSaving%5D%2C%5BRetainDataUntil%5D%20AS%20%5BRetainDataUntil%5D%2C%5BRetainedBy%5D%20AS%20%5BRetainedBy%5D%2C%5BEncrypted%5D%20AS%20%5BEncrypted%5D&parameter.timeframe=-P3M%20P0D&parameter.IsJobAged%5B%5D=0&parameter.datasource%5B%5D=2&limit=20

My request looks like this: http://:81/SearchSvc/CVWebService.svc/CustomReportsEngine/rest/reportsplusengine/datasets/695DADC9-BD89-4B86-B294-AF04D175FD06:1151d30e-fd18-4536-b115-73df02b973fc/data {'Host': '', 'Accept': 'application/json', 'Authtoken': 'QSDK 35caf550adea4f77e0cc5aebecb66a7847861d2fba411a6ce5ef591bcde2c6ce1c295f9f6bd0b798f13821158b98e76e7f76659831607f85bc66722df55b295e4564622c5c201f209c25c147bf6af25a42281ed0d0f59e03978414ee1feb74f863aba063193a3cedb7b2a54f4464c1a00ef68043a5c19138ccd5f092c8ae457df01742f15fe56e026578bf060109beaf5d272455d5aa18f18a5947f0dd88548fe917ad6124f4f9add3e134eb3edf89b68fd8a0cbfb121e0a1ab06b40f4b0cbb9fa0a6f683cf65f06a7a40cfc35080d0a0ec6ff851d979d07759d2fcfc964ea7f795c39d58ad42b2c935ad5c54a9e10037d648fb631d79ea7aa12def27462a6059'}

But this seems to fail: <Response [404]>

mohan-a7 commented 2 years ago

Hi @asshutossh

We have a QScript for fetching the job details on the Copy. We are adding a method to StoragePolicyCopy Object in storage_policy.py to use this QScript and fetch details of jobs on Copy.

While the change goes through regular process of review and test prior to Releasing, you can use the following code snippet meanwhile.

from cvpysdk.commcell import Commcell
from cvpysdk.exception import SDKException

def get_jobs_on_copy(copy_obj, from_date=None, to_date=None, backup_type=None, retained_by=0,
                     include_to_be_copied_jobs=False, list_partial_jobs_only=False):
    """Fetches the Details of jobs on Storage Policy Copy

    Args:
        copy_obj       (Object): Storage Policy Copy object

        from_date      (str): Start Date Range for the Jobs
                                [format-'yyyy/mm/dd'] [default: from start]

        to_date        (str): End Date Range for the Jobs
                                [format-'yyyy/mm/dd'] [default: till date]

        backup_type    (str): Filter by backup type
                                [possible values: 'full'/'incr'] [default: None(all backup types)]

        retained_by    (int): Filter by retention type of jobs
                                [possible values: 1(basic retention)/2(extended retention)/4(manual retention)]
                                [default: 0(all retention types)]

        include_to_be_copied_jobs   (bool): Include details on jobs that are in to be copied state
                                            [possible values: True/False] [default: False]

        list_partial_jobs_only      (bool): Get details of jobs that are in partially copied state only
                                            [possible values: True/false] [default: False]

    Returns:
        (list)  :   List of dict's with each dict containing details of a job
    Raises:
        SDKException:   if the response/fetch operation failed
    """
    command = f"qoperation execscript -sn QS_JobsinSPCopy -si @i_policyName='{copy_obj.storage_policy._storage_policy_name}' -si @i_copyName='{copy_obj.copy_name}'"
    if from_date:
        command = f"{command} -si @i_fromTime='{from_date}'"
    if to_date:
        command = f"{command} -si @i_toTime='{to_date}'"
    if backup_type:
        command = f"{command} -si @i_backupType='{backup_type.lower()}'"
    if retained_by:
        command = f"{command} -si @i_retention='{retained_by}'"
    if include_to_be_copied_jobs:
        command = f"{command} -si @i_includeToBeCopiedJobs='1'"
    if list_partial_jobs_only:
        command = f"{command} -si @i_includePartialJobsOnly='1'"

    response = copy_obj._commcell_object.execute_qcommand(command)
    if response.json():
        json_response = response.json()
        if json_response.get("ExecScriptOutput"):
            if isinstance(json_response.get("ExecScriptOutput").get("FieldValue"), list):
                return json_response.get("ExecScriptOutput").get("FieldValue")
            if isinstance(json_response.get("ExecScriptOutput").get("FieldValue"), dict):
                if json_response.get("ExecScriptOutput").get("FieldValue").get("@JobID"):
                    return [json_response.get("ExecScriptOutput").get("FieldValue")]
            return []
        else:
            response_string = copy_obj._commcell_object._update_response_(response.text)
            raise SDKException('Job', '105', response_string)
    else:
        response_string = copy_obj._commcell_object._update_response_(response.text)
        raise SDKException('Response', '101', response_string)

def fetch_job_ids_list(jobs_details_list):
    """Fetches the Details of jobs on Storage Policy Copy

        Args:
            jobs_details_list (list): List containing details of Jobs
    """
    return [job.get('@JobID') for job in jobs_details_list]

cs = Commcell('cs.testdomain.com', 'username', 'password')

sp = cs.storage_policies.get('TestPolicy')
spc = sp.get_copy('TestCopy')

jobs_details_list = get_jobs_on_copy(spc, include_to_be_copied_jobs=True)
# jobs_details_list has all the details like "jobId, jobType, DataStatus(Status Of Job On Copy), AgentType, AppSize"
job_ids_list = fetch_job_ids_list(jobs_details_list)
# perform your operation (donotcopy/recopy/pickforcopy) using this job_ids_list
print(job_ids_list)
# spc.recopy_jobs(job_ids_list)

Note As you can see, this is using the QScript JobsInSPCopy to fetch the details. However, there is a known issue with this QScript. The DataStatus[Status of Job on that copy] might be wrong for [To-Be-Copied/Partially Copied] Jobs. We will be fixing this as well.