transfer cannot be approved if a data item in the request was invalidated before approval

[tommasoboccali]

Hi, we have a transfer which we cannot approve since weeks, the message being

Apologies, looks like we have an internal server error, details of which below. If the problem persists, please submit a bug report.

Error time=2015-12-04 18:30:08 UTC id=3ca83ab2167326e14f8dce7f91981d6a

Phedex transfer is 525772, link is

https://cmsweb.cern.ch/phedex/prod/Request::View?request=525772

thanks

tom

[nikmagini]

Hi Tommaso,

thanks for reporting. Looking into the server logs for this error id, I see that there is a dataset with NULL ID in the transfer request (1). It seems that one of the datasets in the request was invalidated before approval (2): /BlackHole_BH2_MD-2000_MBH-9000_n-2_TuneCUETP8M1_13TeV-blackmax/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM

Suggested workaround: disapprove the request, then resubmit a new transfer request with the same parameters, excluding this dataset. Informing @vlimant who submitted the original request.

Keeping this ticket open to track the fix for the SQL statement below.

Cheers Nicolo'

1) UpdateRequest: Some bizarre error: DBD::Oracle::st execute failed: ORA-01400: cannot insert NULL into ("CMS_TRANSFERMGMT"."T_DPS_SUBS_DATASET"."DATASET") (DBD ERROR: error possibly near <*> indicator at char 787 in 'merge into t_dps_subs_dataset sd using (select :destination destination, rds.dataset_id dataset, :param param, rx.time_start time_fill_after, rx.is_move, pm.is_cust odial from t_req_request r join t_req_xfer rx on rx.request = r.id join t_req_dataset rds on rds.request = r.id join t_dps_subs_param pm on pm.request = r.id where pm.id = :param ) rd on (rd .destination = sd.destination and (rd.dataset = sd.dataset)) when matched then update set sd.param = rd.param , sd.time_fill_after = rd.time_fill_after, sd.is_move = rd.is_move where rd.is_c ustodial=(select is_custodial from t_dps_subs_param where id=sd.param) and (sd.is_move = 'n' or sd.is_move = rd.is_move) when not matched then insert (destination, dataset, param, time_fill_ after, is_move, time_create) values (rd.destination, <*>rd.dataset, rd.param, rd.time_fill_after, rd.is_move, :time_create)') [for Statement "merge into t_dps_subs_dataset sd using (select : destination destination, rds.dataset_id dataset, :param param, rx.time_start time_fill_after, rx.is_move, pm.is_custodial from t_req_request r join t_req_xfer rx on rx.request = r.id join t_ req_dataset rds on rds.request = r.id join t_dps_subs_param pm on pm.request = r.id where pm.id = :param ) rd on (rd.destination = sd.destination and (rd.dataset = sd.dataset)) when matched then update set sd.param = rd.param , sd.time_fill_after = rd.time_fill_after, sd.is_move = rd.is_move where rd.is_custodial=(select is_custodial from t_dps_subs_param where id=sd.param) and (sd.is_move = 'n' or sd.is_move = rd.is_move) when not matched then insert (destination, dataset, param, time_fill_after, is_move, time_create) values (rd.destination, rd.dataset, rd.param, rd.time_fill_after, rd.is_move, :time_create)" with ParamValues: :destination='8', :param='382201', :time_create=1449245253] at /data/srv/beHG1512e/sw.pre/slc6_amd64_gcc481/cms/PHEDEX-datas vc/2.3.21-comp6/perl_lib/PHEDEX/Core/DB.pm line 322.

2) SQL>select * from t_req_dataset where request=525772 and dataset_id is null; 525772 /BlackHole_BH2_MD-2000_MBH-9000_n-2_TuneCUETP8M1_13TeV-blackmax/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM SQL> select * from t_dps_dataset where name='/BlackHole_BH2_MD-2000_MBH-9000_n-2_TuneCUETP8M1_13TeV-blackmax/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM';

no rows selected

[vlimant]

Hi, how do we see that /BlackHole_BH2_MD-2000_MBH-9000_n-2_TuneCUETP8M1_13TeV-blackmax/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM has been invalidated ? I have no trace of removing it from anywhere. could the one file of the one block had been invalidated by the transfer team ?

[nikmagini]

Hi Jean-Roch,

PhEDEx doesn't keep log of invalidations itself, but invalidation is the only explanation that comes to my mind to explain this issue. (if the dataset had never been injected into PhEDEx in the first place, it would have been skipped in the request instead of causing error)

The transfer team should keep log of their invalidations, let's see if they can find trace of this dataset.

Cheers N.

[nikmagini]

Hi,

file status in DBS is also consistent with invalidation by the Transfer Team:

https://cmsweb.cern.ch/dbs/prod/global/DBSReader/files/?detail=true&dataset=/BlackHole_BH2_MD-2000_MBH-9000_n-2_TuneCUETP8M1_13TeV-blackmax/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM

Cheers N

[joadiazcr]

Hi All,

Several files were invalidated according to ticket 116917[1], there were issues related to gridftp/xrootd at T2_US_Caltech. The file posted by Nicolo is in the following list:(Update#9 in the ticket):

/afs/cern.ch/user/j/jodiazcr/public/116917.list

Regards, Jorge.

[1] https://ggus.eu/?mode=ticket_info&ticket_id=116917