search

ivmfnal / data_dispatcher

BSD 3-Clause "New" or "Revised" License
1 stars 3 forks source link

Does DD sort by preference or do I need to do that #17

Closed hschellman closed 1 year ago

hschellman commented 1 year ago

I'm running at FNAL.

possible replicas {'name': 'np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'namespace': 'pdsp_det_reco', 'path': '/dpm/lancs.ac.uk/home/dune/pdsp_det_reco/13/4c/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'url': 'root://fal-pygrid-30.lancs.ac.uk//dpm/lancs.ac.uk/home/dune/pdsp_det_reco/13/4c/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'rse': 'LANCASTER', 'preference': 100, 'available': True, 'rse_available': True}

{'name': 'np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'namespace': 'pdsp_det_reco', 'path': '/pnfs/fnal.gov/usr/dune/tape_backed/dunepro/protodune-sp/full-reconstructed/2020/detector/physics/PDSPProd4/00/00/52/76/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'url': 'root://fndca1.fnal.gov:1094//pnfs/fnal.gov/usr/dune//tape_backed/dunepro/protodune-sp/full-reconstructed/2020/detector/physics/PDSPProd4/00/00/52/76/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'rse': 'FNAL_DCACHE', 'preference': 0, 'available': True, 'rse_available': True} found a valid replica at root://fal-pygrid-30.lancs.ac.uk//dpm/lancs.ac.uk/home/dune/pdsp_det_reco/13/4c/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root Replica: {'name': 'np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'namespace': 'pdsp_det_reco', 'path': '/dpm/lancs.ac.uk/home/dune/pdsp_det_reco/13/4c/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'url': 'root://fal-pygrid-30.lancs.ac.uk//dpm/lancs.ac.uk/home/dune/pdsp_det_reco/13/4c/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root', 'rse': 'LANCASTER', 'preference': 100, 'available': True, 'rse_available': True}

found uri root://fal-pygrid-30.lancs.ac.uk//dpm/lancs.ac.uk/home/dune/pdsp_det_reco/13/4c/np04_raw_run005276_0001_dl8_reco1_14390424_0_20201121T070447Z.root

shows preference=0 for FNAL, 100 for lancaster but it seems to choose lancaster as it is first. Should I assume I need to do the sort?

hschellman commented 1 year ago

Adding this code to my code made it work

instead of for r in replicas:

    pref,index = 1000,0
    count = 0
    for r in replicas:

        if r["preference"] < pref:
            pref,index =r["preference"],count
            print ("found a better rse",r["rse"],pref,index,count)
        count +=1
    for r in [replicas[index]]:
ivmfnal commented 1 year ago

DD API does not sort replicas because it returns replicas as an unsorted dictionary indexed by the RSE name. Each entry does have a numeric preference, which can be used by the client to choose optimal replica.

"dd worker next" UI command does sort the output and how it is fixed to print replicas in the order of increasing preferences