DIRACGrid / DIRAC

DIRAC Grid
http://diracgrid.org
GNU General Public License v3.0
114 stars 175 forks source link

More configuration flexibility in InputDataPolicy #6820

Open panta-123 opened 1 year ago

panta-123 commented 1 year ago

Some additional flexibility will be good to add.

  1. Choose InputDataResolution policy according to jobType specified in CS. -> Lets say i want only User job to use InputDataByProtocol and other job to use DownloadInputData

  2. If there is no locol protcol that we want to use for streaming fallback to download policy. -> Curently for streaming p[rotocols we can specify Local and Remote protocols as:

       # List of protocols to use for streaming
      Protocols
      {
        # This list is used if the we are getting a file from a
        # StorageElement local to the site we are running on
        Local = file, xroot, root
        # This list is used if the SE is not local
        Remote = xroot, root
      }

    Lets say if the Remote part is empty and no local protocol available, then use the default policy.(i.e. DownloadInputData)

chaen commented 1 year ago

These are nice ideas, and we welcome PR ! The input data resolution being based on plugins, it should not be too difficult for you to extend. You should find a good starting point here https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/InputDataResolution.html?highlight=inputdatapolicy Also, what you offer seems generic enough, and would probably benefit to other communities, so please consider implementing this in vanilla DIRAC. Let us know if you need more guidance.