distributed intermediary storage

[dsschult]

(imported from trac: #1580)

Based on where the node we are running on is located, choose an ideal intermediary storage location.

This probably needs:

• distance graph between various clusters and storage locations
  • storage for this in the DB
  • cron task to take latest task statistics and update the graph
• ability to find location on that graph
  • use external domain name?
• algorithm for choosing storage based on distance (and maybe load?)
  • this is probably an rpc call to the server, so we don't lug around the distance graph

These storage locations also need to be set up and accept our gridftp certificates (using icecube VO?). Probably a job for @gonzalomerino.

[gonzalomerino]

David,

For the moment, we have only two locations: gridftp at UW and gridftp at DESY.

My suggestion would be to reduce the scope of this for the first versions of iceprod2.

What I think will be very useful at the beginning will be that iceprod2 jobs have the capability of using desy to store intermediate files, if the UW madison storage fails.

A full blown multi-storage mesh where network distances are optimized, etc, looks nice but I see that more for the future.