search

scikit-hep / uproot5

ROOT I/O in pure Python and NumPy.
https://uproot.readthedocs.io
BSD 3-Clause "New" or "Revised" License
232 stars 73 forks source link

Support XRootD URLs with multiple servers. #5

Open chrisburr opened 4 years ago

chrisburr commented 4 years ago

From the TXNetFile docs:

The "url" argument must be of the form

root://server1:port1[,server2:port2,...,serverN:portN]/pathfile,

Note that this means that multiple servers (>= 1) can be specified in the url. The connection will try to connect to the first server:port and if that does not succeed, it will try the second one, and so on until it finds a server that will respond.

I didn't realise this was an pre-existing feature in ROOT but I'd been asking for it for a while. Being able to do this better matches with how data is stored on the grid and makes it easier to when sites are down.

The implementation of this shouldn't only check that the server is alive, it should also fallback if errors are returned when trying to access the data (i.e. only querying the config isn't enough to know if the site is down).

jpivarski commented 4 years ago

Are the fallbacks accessible in the XRootD.client.URL object?

Recovering from a nonexistent URL takes quite a while. The timeout has to be an integer number of seconds and passing "1" means as much as 5‒10 seconds in practice. (Eventually, I'll be turning off unit test #1, or at least making it part of a "long tests" suite.)

jpivarski commented 7 months ago

@lobis, @nsmith-, is this something that can be done with fsspec-xrootd now?

It could be cool to finish off an issue with a low number like #5!

nsmith- commented 7 months ago

I think this ties very closely with https://github.com/CoffeaTeam/fsspec-xrootd/issues/36 The implementations would share a lot of the same logic w.r.t. keeping open connections to multiple servers and ensuring we use the best server for a file at any given time.