WMCore/Storage/StageOutImpl.py should spit out stdout on line 65 when it raises

PerilousApricot commented 12 years ago

It's nice to have the command executed, but it really needs to have what the command returned. Also, this could stand to be cleaned up a lot.

{{{

Job 761 -retry: 1 ----------------- failure details for step 'stageOut1': Failure type: Misc. StageOut error: 99109 Detailed err: Error in StageOut: 99109 StageOutFailure Message: Failure for fallback stage out: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 187, in call pfn = self.localStageOut(lfn, fileToStage['PFN']) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 308, in localStageOut TargetPFN = pfn) StageOutFailure: StageOutFailure Message: Failure for local stage out: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: None

Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 297, in localStageOut impl(protocol, localPfn, pfn, options) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 214, in call raise ex StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: None

TargetPFN : srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
ErrorCode : 60311
ModuleName : WMCore.Storage.StageOutError
MethodName : __init__
LFN : /store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
ClassInstance : None
FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py
ClassName : None
Command : srm
LineNumber : 32
InputPFN : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root
Protocol : srm
ErrorNr : 0
ErrorType : GeneralStageOutFailure

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 297, in localStageOut impl(protocol, localPfn, pfn, options) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 214, in call raise ex StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: None

TargetPFN : srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
ErrorCode : 60311
ModuleName : WMCore.Storage.StageOutError
MethodName : __init__
LFN : /store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
ClassInstance : None
FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py
ClassName : None
Command : srm
LineNumber : 32
InputPFN : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root
ErrorNr : 0
ErrorType : GeneralStageOutFailure

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 258, in fallbackStageOut impl(fbParams['command'], localPfn, pfn, fbParams.get("option", None)) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 214, in call raise ex StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://dc2-grid-64.brunel.ac.uk:8443/srm/managerv1?SFN=/dpm/brunel.ac.uk/home/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 187, in call pfn = self.localStageOut(lfn, fileToStage['PFN']) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 308, in localStageOut TargetPFN = pfn) StageOutFailure: StageOutFailure Message: Failure for local stage out: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: None

Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutMgr.py", line 297, in localStageOut impl(protocol, localPfn, pfn, options) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 214, in call raise ex StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: Traceback (most recent call last): File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 196, in call self.executeCommand(command) File "/srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh REPORT_FILE=pwd/srm.report.$$ srmcp -report=$REPORT_FILE -retry_num=0 -debug -streams_num=1 file:////srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root

        EXIT_STATUS=`cat $REPORT_FILE | cut -f3 -d" "`
        echo "srmcp exit status: $EXIT_STATUS"
        if [[ $EXIT_STATUS != 0 ]]; then
           echo "Non-zero srmcp Exit status!!!"
           echo "Cleaning up failed file:"
           srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root
           exit 60311
        fi

        FILE_SIZE=`stat -c %s //srv/localstage/scratch/4604372.1.grid.q/CREAM917627004/job/WMTaskSpace/cmsRun1/out.root `

echo "Local File Size is: $FILE_SIZE"

    for ((a=1; a <= 10 ; a++))
    do
       SRM_SIZE=`srm-get-metadata -retry_num=0 srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 2>/dev/null | grep 'size :[0-9]' | cut -f2 -d":"`
       echo "SRM Size is $SRM_SIZE"
       if [[ $SRM_SIZE > 0 ]]; then
          if [[ $SRM_SIZE == $FILE_SIZE ]]; then
             exit 0
          else
             echo "Error: Size Mismatch between local and SE"
             echo "Cleaning up failed file:"
             srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
             exit 60311
          fi 
       else
          sleep 2
       fi
    done
    echo "Cleaning up failed file:"
    srm-advisory-delete srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv1?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/temp/user/meloam/BTag/1319812005/v4/0000/60A73A34-7009-E111-82FD-842B2B7680A2.root 
    exit 60311

LineNumber : 32
ErrorNr : 0
ExitCode : 151

Traceback: None

TargetPFN : srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/

PerilousApricot commented 12 years ago

meloam: Try this delicious patch:

{{{

diff --git a/src/python/WMCore/Storage/StageOutImpl.py b/src/python/WMCore/Storage/StageOutImpl.py index cbcbf00..6f51825 100644 --- a/src/python/WMCore/Storage/StageOutImpl.py +++ b/src/python/WMCore/Storage/StageOutImpl.py @@ -8,7 +8,7 @@ inherit this object and implement the methods accordingly """ import time import os -from WMCore.Storage.Execute import runCommandWithOutput +from WMCore.Storage.Execute import runCommand from WMCore.Storage.StageOutError import StageOutError, StageOutInvalidPath

class StageOutImpl: @@ -51,7 +51,7 @@ class StageOutImpl:

     """
     try:

exitCode,output = runCommandWithOutput(command)
exitCode = runCommand(command) msg = "Command exited with status: %s" % (exitCode) print msg except Exception, ex: @@ -62,7 +62,7 @@ class StageOutImpl: msg = "Command exited non-zero" print "ERROR: Exception During Stage Out:\n" print msg
raise StageOutError("%s\nOutput was: %s" % (msg, output) , Command = command, ExitCode = exitCode)
raise StageOutError(msg, Command = command, ExitCode = exitCode) return

}}}

Also, this will be super handy for CRAB3, so could we try and do this fast to make it into the next round of testing?

sfoulkes commented 12 years ago

sfoulkes: It looks like your patch is backwards.

evansde77 commented 12 years ago

evansde: 100% Guarantee this is getting bumped to March...

sfoulkes commented 12 years ago

sfoulkes: Fixed by the new stageout code, whenever that happens. Closing.

dmwm / WMCore

WMCore/Storage/StageOutImpl.py should spit out stdout on line 65 when it raises #2793