search

ssl-hep / ServiceX

ServiceX - a data delivery service pilot for IRIS-HEP DOMA
BSD 3-Clause "New" or "Revised" License
20 stars 21 forks source link

ServiceX DID-finder stalls for Rucio dataset accessed via https protocol #539

Closed kyungeonchoi closed 1 year ago

kyungeonchoi commented 1 year ago

Describe the bug ServiceX DID-finder stalls for Rucio dataset accessed via https protocol. A ServiceX request seems to wait for the file list returned from DID-finder. Once Rucio dataset is moved to MWT2 (xrood protocol) ServiceX can get the file list and runs fine.

An example Rucio RSE for https protocol: NEVIS_GRIDFTP: https://xenia.nevis.columbia.edu:1094/atlas/dq2/rucio/user/dpanchal/be/a4/user.dpanchal.31107845._000054.hist-output.root

ivukotic commented 1 year ago

Well, I would not call it a bug... All of the ATLAS storages are required to provide an xroot endpoint. So this is an issue of the Nevis storage. On the other hand we would like to be able to use (and also cache) data read over http. I am working on it, and if/when performance is good enough, I will make it possible to use both xroot and http.

ponyisi commented 1 year ago

Hi Ilija - understood that it's an issue with Nevis not necessarily supporting the "proper" endpoint, but I consider it a problem if we cannot access a file that is available via "rucio get" (as this one is). KyungEon, you reported a site that was webdav ?

kyungeonchoi commented 1 year ago

Thanks @ivukotic for looking into this problem. I didn't know that http doesn't need to be a supported endpoint. As you said it would be still good to support both xrootd and http. @ponyisi - I will talk to Dev.

ivukotic commented 1 year ago

Hi,

I just started work on getting xcaches to be able to use paths like root://xcache.xx.edu//http://origin.xx.edu/…

Ilija

From: KyungEon Choi @.> Sent: Sunday, February 19, 2023 20:07 To: ssl-hep/ServiceX @.> Cc: Ilija Vukotic @.>; Mention @.> Subject: Re: [ssl-hep/ServiceX] ServiceX DID-finder stalls for Rucio dataset accessed via https protocol (Issue #539)

Thanks @ivukotichttps://urldefense.com/v3/__https:/github.com/ivukotic__;!!BpyFHLRN4TMTrA!74EHO7V3VhZHP-PvE7iPuqai_esC6DW3nMOMqqEEmy8j8HROiLPmQzzBUNV3ZoEyB0ElOX6ESe-EhXu4-bpNJ2RHCQ3r$ for looking into this problem. I didn't know that http doesn't need to be a supported endpoint. As you said it would be still good to support both xrootd and http. @ponyisihttps://urldefense.com/v3/__https:/github.com/ponyisi__;!!BpyFHLRN4TMTrA!74EHO7V3VhZHP-PvE7iPuqai_esC6DW3nMOMqqEEmy8j8HROiLPmQzzBUNV3ZoEyB0ElOX6ESe-EhXu4-bpNJxDnv_ZJ$ - I will talk to Dev.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/ssl-hep/ServiceX/issues/539*issuecomment-1436197095__;Iw!!BpyFHLRN4TMTrA!74EHO7V3VhZHP-PvE7iPuqai_esC6DW3nMOMqqEEmy8j8HROiLPmQzzBUNV3ZoEyB0ElOX6ESe-EhXu4-bpNJ-OVcyoL$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AANICGAJYHSZXG32DTJK3YDWYLGUVANCNFSM6AAAAAAUXM3KT4__;!!BpyFHLRN4TMTrA!74EHO7V3VhZHP-PvE7iPuqai_esC6DW3nMOMqqEEmy8j8HROiLPmQzzBUNV3ZoEyB0ElOX6ESe-EhXu4-bpNJzMTazTq$. You are receiving this because you were mentioned.Message ID: @.**@.>>