search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
285 stars 136 forks source link

(General problem: Unable to find address that faces lxplus727.cern.ch/2001:1458:d00:14:0:0:0:63 [666]) #6568

Closed cfgamboa closed 2 years ago

cfgamboa commented 2 years ago

Hello,

I am trying to understand why xrdcp transfer fails from lxplus but it succeeds from a worker node at BNL. If I use GridFTP or DAVs the transfer succeeds from CERN.

[cgamboa@lxplus727 ~]$ gfal-copy -v -f file:////tmp/test.cern root://dcdoor19.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1
Copying 35674 bytes file:////tmp/test.cern => root://dcdoor19.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1
event: [1648614433962] BOTH   GFAL2:CORE:COPY   LIST:ENTER  
event: [1648614433962] BOTH   GFAL2:CORE:COPY   LIST:ITEM   file:////tmp/test.cern => root://dcdoor19.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1
event: [1648614433962] BOTH   GFAL2:CORE:COPY   LIST:EXIT   
event: [1648614433963] BOTH   xroot TRANSFER:ENTER  file://localhost///tmp/test.cern?xrd.gsiusrpxy=/tmp/x509up_u18256&xrdcl.intent=tpc => root://dcdoor19.usatlas.bnl.gov:1096///pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1?xrd.gsiusrpxy=/tmp/x509up_u18256&xrdcl.intent=tpc
event: [1648614433964] BOTH   xroot TRANSFER:TYPE   streamed
event: [1648614437222] BOTH   xroot TRANSFER:EXIT   Job finished, [ERROR] Server responded with an error: [3012] Failed to open file (General problem: Unable to find address that faces lxplus727.cern.ch/2001:1458:d00:14:0:0:0:63 [666]) (destination)

gfal-copy error: 14 (Bad address) - Error on XrdCl::CopyProcess::Run(): [ERROR] Server responded with an error: [3012] Failed to open file (General problem: Unable to find address that faces lxplus727.cern.ch/2001:1458:d00:14:0:0:0:63 [666]) (destination)

Logs at the door

level=INFO ts=2022-03-30T00:28:16.613-0400 event=org.dcache.xrootd.connection.start session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA socket.remote=[2001:1458:d00:14::63]:40970 socket.local=[2620:0:210:1::f2]:1096
level=INFO ts=2022-03-30T00:28:16.626-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=protocol response=ok
level=INFO ts=2022-03-30T00:28:16.718-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=login username=cgamboa capver=5 pid=25501 token=?xrd.cc=ch&xrd.tz=1&xrd.appname=python2.7&xrd.info=&xrd.hostname=lxplus727.cern.ch&xrd.rn=v5.4.2 response=ok sessionId=22FEE7E3D1F706EDE2B37BE6E51ABA35 sec=&P=gsi,v:10400,c:ssl,ca:f5f0dfc2
level=INFO ts=2022-03-30T00:28:16.864-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=auth response=authmore
level=INFO ts=2022-03-30T00:28:17.000-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=auth response=authmore
level=INFO ts=2022-03-30T00:28:17.126-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=auth response=ok
level=ERROR ts=2022-03-30T00:28:17.224-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=stat path=//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1 opaque=xrd.gsiusrpxy=/tmp/x509up_u18256 vfs=false response=error error.code=NotFound error.msg="No such file"
level=ERROR ts=2022-03-30T00:28:17.326-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=stat path=//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1 opaque=xrd.gsiusrpxy=/tmp/x509up_u18256 vfs=false response=error error.code=NotFound error.msg="No such file"
level=INFO ts=2022-03-30T00:28:17.511-0400 event=org.dcache.xrootd.connection.start session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA socket.remote=[2001:1458:d00:14::63]:40996 socket.local=[2620:0:210:1::f2]:1096
level=INFO ts=2022-03-30T00:28:17.527-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA request=protocol response=ok
level=INFO ts=2022-03-30T00:28:17.619-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA request=login username=cgamboa capver=5 pid=25501 token=?xrd.cc=ch&xrd.tz=1&xrd.appname=python2.7&xrd.info=&xrd.hostname=lxplus727.cern.ch&xrd.rn=v5.4.2 response=ok sessionId=B5663249996E70CFE160445BE627842A sec=&P=gsi,v:10400,c:ssl,ca:f5f0dfc2
level=INFO ts=2022-03-30T00:28:17.744-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA request=auth response=authmore
level=INFO ts=2022-03-30T00:28:17.889-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA request=auth response=authmore
level=INFO ts=2022-03-30T00:28:17.990-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA request=auth response=ok
level=ERROR ts=2022-03-30T00:28:18.565-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA request=open path=//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1 opaque=oss.asize=35674&xrd.gsiusrpxy=/tmp/x509up_u18256 mode=0644 options=0x1462 response=error error.code=ServerError error.msg="Failed to open file (General problem: Unable to find address that faces lxplus727.cern.ch/2001:1458:d00:14:0:0:0:63 [666])"
level=ERROR ts=2022-03-30T00:28:18.668-0400 event=org.dcache.xrootd.request session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA request=rm path=//pnfs/usatlas.bnl.gov/atlasgroupdisk/SAM/test.carlos.1 opaque=xrd.gsiusrpxy=/tmp/x509up_u18256 response=error error.code=NotFound error.msg="No such file"
level=INFO ts=2022-03-30T00:28:18.798-0400 event=org.dcache.xrootd.connection.end session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/l+4nA
level=INFO ts=2022-03-30T00:28:18.798-0400 event=org.dcache.xrootd.connection.end session=door:Xrootd-dcdoor19@xrootd-dcdoor19Domain:AAXbZ/lxLqA

All the best, Carlos

cfgamboa commented 2 years ago

Hello,

Similar to what could be done for DAV protocol, is there any way to enable the XROOT door to specify an internal network to interact with internal network isolated resources?

I could not find a specific one similar to webdav.net.internal

All the best, Carlos

cfgamboa commented 2 years ago

Another particularity of this transfers is that, the XROOT doors tries to get directly to the pool (internal) instead of using the DMZ pool. These are other logs from a transfer from CERN to BNL.

Same type of transfer works fine for workers nodes at BNL, they use the internal interface to interact with the door.

This is an extract of the of the door log 

31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] WRITE: redirect[dc207.usatlas.bnl.gov:33151,tried=dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov&org.dcache.xrootd.client=cgamboa.19834@lxplus712.cern.ch&org.dcache.uuid=8e318f41-ada5-47fc-b678-6b2151e9a589&triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,]
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] FLUSH
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] READ: open[0,1104,/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1,org.dcache.uuid=8e318f41-ada5-47fc-b678-6b2151e9a589&org.dcache.xrootd.client=cgamboa.19834@lxplus712.cern.ch&tried=dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov&triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr]
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] READ COMPLETE
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] open[0,1104,/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1,org.dcache.uuid=8e318f41-ada5-47fc-b678-6b2151e9a589&org.dcache.xrootd.client=cgamboa.19834@lxplus712.cern.ch&tried=dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov&triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr] –– not a third-party request.
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Opening /pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1 for read
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] open flags: options to apply for open path (raw=1104 ): kXR_async kXR_open_read kXR_retstat
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] mode to apply to open path: --- --- ---
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] OPAQUE : {tried=dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.19834@lxplus712.cern.ch, org.dcache.uuid=8e318f41-ada5-47fc-b678-6b2151e9a589, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] tried hosts : []
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Status: PnfsManager: Fetching storage info
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Status: PoolManager: Selecting pool
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Status: Pool PoolName=dc207_12 PoolAddress=dc207_12@dc207twelveDomain: Creating mover
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg dc207_12 XrootdDoorAdressInfo] Received redirect msg from mover
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Status: Mover PoolName=dc207_12 PoolAddress=dc207_12@dc207twelveDomain/134551399: Waiting for redirect
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Status: Mover PoolName=dc207_12 PoolAddress=dc207_12@dc207twelveDomain/134551399: Sending
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Redirecting to dc207.usatlas.bnl.gov, dc207.usatlas.bnl.gov/10.42.38.11:33151
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Sending the following host information to the client: dc207.usatlas.bnl.gov
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] WRITE: redirect[dc207.usatlas.bnl.gov:33151,tried=dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc223.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov,dc207.usatlas.bnl.gov&org.dcache.xrootd.client=cgamboa.19834@lxplus712.cern.ch&org.dcache.uuid=15c32fd4-c92b-4de3-baa6-bb3028f1cf11&triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,]
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] FLUSH
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 - R:/137.138.125.201:59888] READ COMPLETE
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 ! R:lxplus712.cern.ch/137.138.125.201:59888] INACTIVE
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] [id: 0x7e21e62d, L:/192.12.15.228:1096 ! R:lxplus712.cern.ch/137.138.125.201:59888] UNREGISTERED
31 Mar 2022 11:45:47 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg dc207_12 DoorTransferFinished 000066AD211D2E21459EB8E0770AD1F609E1] Transfer 000066AD211D2E21459EB8E0770AD1F609E1@PoolName=dc207_12 PoolAddress=dc207_12@dc207twelveDomain failed: No connection from client after 300 seconds. Giving up. (error code=10006)
31 Mar 2022 11:45:47 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg dc223_6 DoorTransferFinished 000066AD211D2E21459EB8E0770AD1F609E1] Transfer 000066AD211D2E21459EB8E0770AD1F609E1@PoolName=dc223_6 PoolAddress=dc223_6@dc223sixDomain failed: No connection from client after 300 seconds. Giving up. (error code=10006)

This one seen at the billingdb

3.31 11:45:49 [pool:dc207_12:transfer] [000066AD211D2E21459EB8E0770AD1F609E1,633213989] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1] bnlt0d1:BNLT0D1@osm 0 300000 false {Xrootd-5.0:137.138.125.201:59888} [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg:1648741249518000] {10006:"No connection from client after 300 seconds. Giving up."}
03.31 11:45:49 [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:request] ["/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cgamboa/CN=658196/CN=Carlos Fernando Gamboa":9102:31152:137.138.125.201] [000066AD211D2E21459EB8E0770AD1F609E1,633213989] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1] bnlt0d1:BNLT0D1@osm 300085 0 {10006:"No connection from client after 300 seconds. Giving up."}
03.31 11:45:49 [pool:dc207_12:transfer] [000066AD211D2E21459EB8E0770AD1F609E1,633213989] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1] bnlt0d1:BNLT0D1@osm 0 300000 false {Xrootd-5.0:137.138.125.201:59888} [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg:1648741249710000] {10006:"No connection from client after 300 seconds. Giving up."}
03.31 11:45:49 [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:request] ["/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cgamboa/CN=658196/CN=Carlos Fernando Gamboa":9102:31152:137.138.125.201] [000066AD211D2E21459EB8E0770AD1F609E1,633213989] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1] bnlt0d1:BNLT0D1@osm 300050 0 {10006:"No connection from client after 300 seconds. Giving up."}
03.31 11:45:49 [pool:dc207_12:transfer] [000066AD211D2E21459EB8E0770AD1F609E1,633213989] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1] bnlt0d1:BNLT0D1@osm 0 300000 false {Xrootd-5.0:137.138.125.201:59888} [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg:1648741249854000] {10006:"No connection from client after 300 seconds. Giving up."}
03.31 11:45:49 [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:request] ["/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cgamboa/CN=658196/CN=Carlos Fernando Gamboa":9102:31152:137.138.125.201] [000066AD211D2E21459EB8E0770AD1F609E1,633213989] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/86/00/DAOD_PHYS.28121460._000008.pool.root.1] bnlt0d1:BNLT0D1@osm 300043 0 {10006:"No connection from client after 300 seconds. Giving up."}
cfgamboa commented 2 years ago

Dcache doors on dcache-7.2.7-1.noarch and pools on dcache-7.2.3

alrossi commented 2 years ago

Hi Carlos,

No, unfortunately the xrootd door has no knowledge of how to override the dCache linkage.

I do not know why GridFTP works, but it seems to me this might be a pool manager configuration issue. The Xrootd door is getting the pool from the pool manager. The pool manager should not be giving a pool that should be inaccessible to the outside network to the connecting client to use. That suggests to me that the pool selection unit setup is incorrect.

I remember a while back helping BNL with some issues (Third Party Copy)? and it seemed to me xrootd ultimately had no issues with the BNL setup (internal vs edge network).

In general what needs to be done (and what I thought BNL does) is make the DMZ read/writable to external, and then link them to the internal pools which should not be. This is all done through the dcache setup as I said. I don't know if this has changed. When a client writes or reads to the DMZ pool, it is p2p copied under to covers to the internal pools.

Perhaps Iris (Yingzi) Wu or Jane Liu can help out here?

iriswu@bnl.gov zhliu@rcf.rhic.bnl.gov

Al

cfgamboa commented 2 years ago

Hello Al

Thank you. Not only GFTP works (though it using its own adapters that handle the internal interface) but DAVS works fine and it uses the DMZ pools. XROOT does not seem to be able to use the DMZ pools though as you also noticed.

All the best, Carlos

On Apr 1, 2022, at 2:38 PM, Albert Rossi @.***> wrote:

Hi Carlos,

No, unfortunately the xrootd door has no knowledge of how to override the dCache linkage.

I do not know why GridFTP works, but it seems to me this might be a pool manager configuration issue. The Xrootd door is getting the pool from the pool manager. The pool manager should not be giving a pool that should be inaccessible to the outside network to the connecting client to use. That suggests to me that the pool selection unit setup is incorrect.

I remember a while back helping BNL with some issues (Third Party Copy)? and it seemed to me xrootd ultimately had no issues with the BNL setup (internal vs edge network).

In general what needs to be done (and what I thought BNL does) is make the DMZ read/writable to external, and then link them to the internal pools which should not be. This is all done through the dcache setup as I said. I don't know if this has changed.

Perhaps Iris (Yingzi) Wu or Jane Liu can help out here?

Al

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6568#issuecomment-1086216563, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO5J6MWK5JCZRW4IIWDVC47CXANCNFSM5SAUAKRQ. You are receiving this because you authored the thread.

alrossi commented 2 years ago

Hi Carlos,

Well, xrootd does not have those adapters available to it (at least as far as I know). To get xrootd to work, you need to rely on the pool setup, as I said above.

I can't say much more about this without input from BNL. I suggest speaking with them. There must be other xrootd users. Are they having similar issues?

Cheers, Al

cfgamboa commented 2 years ago

==== Hello Al,

I am using my user hat to test, but I am also “recently" helping in supporting the dCache infrastructure at BNL. I looked briefly at the configuration of the pool manager and even attempted to run a simple test using different reference like this one (https://github.com/dCache/dcache/blob/6a2bc7a0ff3a9d14fb6a381b0643c9b1c03219f0/docs/TheBook/src/main/markdown/config-PoolManager.md#the-pool-selection-mechanism)

Please note that internal transfers using XROOT door do not have this issue.

Unfortunately the DMZ pool is not used by the Xroot door. But even in that situation the internal pool will need to be told to use the internal interface.

I assume that you did not observed something unusual on the logs.

Thank you for advising,

Carlos

cfgamboa commented 2 years ago

Hello @alrossi

Just to confirm that the xrootd/* is enabled. The remote-transfer-protocol is also use by DAVs to access to the DMZ, no issues observed for DAVs

[dcadmin02] (PoolManager@dCacheDomain) admin > psu ls ugroup  remote-transfer-protocol  -l 
remote-transfer-protocol
 unitList :
   RemoteHttpsDataTransfer/1  (type=PROTOCOL;canonical=RemoteHttpsDataTransfer/1;uGroups=1)
   xrootd/*  (type=PROTOCOL;canonical=xrootd/*;uGroups=1)

All the best, Carlos

alrossi commented 2 years ago

Carlos,

would it be possible for you to attach the poolmanager.conf file for the dCache installation that Xrootd-dcdoor07 is part of?

I am unable to understand what might be the issue here on the basis of the logs you have attached.

This:

Status: PoolManager: Selecting pool
31 Mar 2022 11:40:49 (Xrootd-dcdoor07) [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXbhXwnVNg] Status: Pool PoolName=dc207_12 PoolAddress=dc207_12@dc207twelveDomain: Creating mover

is not something the xrootd door is doing. It is the pool manager that has handed off the pool to the door to redirect the client to (i.e., it is not the door deciding what pool to use).

If you used some other protocol besides xrootd, it would do the same thing, except if it has the ability to proxy the connection through the door. GridFTP does. HTTPS does. But xrootd does not.

My only guess is that somehow the DMZ pools are linked to network unit improperly: the pool manager believes that the incoming connection can read from that pool when in reality it cannot because of firewalls, etc.

The tests I was referring to above were using xrootd from SLAC to BNL. BNL was interested in being part of the third-party-copy testing conducted by WLCG up until this FALL for xrootd and https third-party. At one point there was an issue with making this work with the BNL setup. I remember emulating the dmz/internal dichotomy on my testbed. If you arrange it correctly, there should be no problem with xrootd. So I continue to suspect that the issue is in the configuration.

Of course, GridFTP and HTTPS don't need that because they are doing something else.

This is only an 'xrootd' problem insofar as xrootd does not provide for proxying a transfer through the door. (It never has.)

Cheers,

Al

cfgamboa commented 2 years ago

Hello @alrossi,

Test [cgamboa@lxplus760 ~]$ xrdcp -v -f file:////tmp/test.cern root://192.12.15.228:1096//pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 [0B/0B][100%][==================================================][0B/s]
Run: [FATAL] Redirect limit has been reached: (destination)

At the door: dcdoor07, X

(Xrootd-dcdoor07@xrootd-dcdoor07Domain) admin > show pinboard 1000

04 Apr 2022 14:57:46 [xrootd-net-55] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Time since last server cert refresh 94272
04 Apr 2022 14:57:46 [xrootd-net-55] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Loading server certificates. Current refresh interval: 60000 ms
04 Apr 2022 14:57:46 [xrootd-net-55] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Verifying host certificate
04 Apr 2022 14:57:47 [xrootd-net-55] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Client protocol version was 10600, using GSIPost49ServerRequestHandler.
04 Apr 2022 14:57:47 [xrootd-net-55] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Time since last server cert refresh 1025
04 Apr 2022 14:57:48 [xrootd-net-55] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Using padded DH secret generation.
04 Apr 2022 14:57:48 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Xrootd-Error-Response: ErrorNr=3011 ErrorMsg=No such file
04 Apr 2022 14:57:48 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:48 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {oss.asize=5}
04 Apr 2022 14:57:48 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc212.usatlas.bnl.gov, dc212.usatlas.bnl.gov/10.42.38.95:33123
04 Apr 2022 14:57:48 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc212.usatlas.bnl.gov
04 Apr 2022 14:57:48 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:48 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=732ed316-5dda-4f8f-86ba-5c688b091008, oss.asize=5, triedrc=srverr}
04 Apr 2022 14:57:49 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc052.usatlas.bnl.gov, dc052.usatlas.bnl.gov/10.42.38.36:33284
04 Apr 2022 14:57:49 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc052.usatlas.bnl.gov
04 Apr 2022 14:57:49 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:49 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=67d15992-835e-4735-b356-cad4ce2b72d1, oss.asize=5, triedrc=srverr,srverr}
04 Apr 2022 14:57:49 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc052.usatlas.bnl.gov, dc052.usatlas.bnl.gov/10.42.38.36:33266
04 Apr 2022 14:57:49 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc052.usatlas.bnl.gov
04 Apr 2022 14:57:49 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:49 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=a3743bf8-388c-4d33-bc09-f4c2542b473e, oss.asize=5, triedrc=srverr,srverr,srverr}
04 Apr 2022 14:57:49 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc201.usatlas.bnl.gov, dc201.usatlas.bnl.gov/10.42.38.5:33218
04 Apr 2022 14:57:49 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc201.usatlas.bnl.gov
04 Apr 2022 14:57:49 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:49 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=35649aec-5ed2-4a24-81a2-b0dabd54f430, oss.asize=5, triedrc=srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:50 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc221.usatlas.bnl.gov, dc221.usatlas.bnl.gov/10.42.38.107:33190
04 Apr 2022 14:57:50 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc221.usatlas.bnl.gov
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=20f916d6-f076-4afc-88e0-671c95320231, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc209.usatlas.bnl.gov, dc209.usatlas.bnl.gov/10.42.38.13:33257
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc209.usatlas.bnl.gov
04 Apr 2022 14:57:50 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:50 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=96f8a652-764f-4e34-9ac9-fa318bc9a2ee, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:50 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc226.usatlas.bnl.gov, dc226.usatlas.bnl.gov/10.42.38.112:33161
04 Apr 2022 14:57:50 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc226.usatlas.bnl.gov
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=65381a0f-472f-43d8-99d1-7a79c75e8e40, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc220.usatlas.bnl.gov, dc220.usatlas.bnl.gov/10.42.38.106:33148
04 Apr 2022 14:57:50 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc220.usatlas.bnl.gov
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=cbbe3784-cff0-47ce-b54a-b8bfa9917ac8, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc216.usatlas.bnl.gov, dc216.usatlas.bnl.gov/10.42.38.99:33220
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc216.usatlas.bnl.gov
04 Apr 2022 14:57:51 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:51 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=d2b2cc60-4640-42e3-ac08-c4b4cb22a0f1, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:51 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc213.usatlas.bnl.gov, dc213.usatlas.bnl.gov/10.42.38.96:33228
04 Apr 2022 14:57:51 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc213.usatlas.bnl.gov
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=1d6d16d7-18d7-4684-b679-d0fdeb7b3a6a, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc215.usatlas.bnl.gov, dc215.usatlas.bnl.gov/10.42.38.98:33229
04 Apr 2022 14:57:51 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc215.usatlas.bnl.gov
04 Apr 2022 14:57:52 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:52 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov,dc215.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=bad13b77-9275-49e9-a0f6-9a828f65603c, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:52 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc220.usatlas.bnl.gov, dc220.usatlas.bnl.gov/10.42.38.106:33237
04 Apr 2022 14:57:52 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc220.usatlas.bnl.gov
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov,dc215.usatlas.bnl.gov,dc220.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=1c283184-0609-422e-af4f-92e2c86147b2, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc217.usatlas.bnl.gov, dc217.usatlas.bnl.gov/10.42.38.100:33149
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc217.usatlas.bnl.gov
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov,dc215.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc217.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=ad20537b-d14c-42c8-be1e-b56a30c4ea2d, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc048.usatlas.bnl.gov, dc048.usatlas.bnl.gov/10.42.38.32:33145
04 Apr 2022 14:57:52 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc048.usatlas.bnl.gov
04 Apr 2022 14:57:52 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:52 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov,dc215.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc217.usatlas.bnl.gov,dc048.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=22af27c1-28e1-46c0-a229-65332233d619, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:53 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc212.usatlas.bnl.gov, dc212.usatlas.bnl.gov/10.42.38.95:33238
04 Apr 2022 14:57:53 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc212.usatlas.bnl.gov
04 Apr 2022 14:57:53 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:53 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov,dc215.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc217.usatlas.bnl.gov,dc048.usatlas.bnl.gov,dc212.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=57a4fb30-542b-42c4-8091-9a31a28cd933, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:53 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc226.usatlas.bnl.gov, dc226.usatlas.bnl.gov/10.42.38.112:33227
04 Apr 2022 14:57:53 [pool-7-thread-87463] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc226.usatlas.bnl.gov
04 Apr 2022 14:57:53 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Opening /pnfs/usatlas.bnl.gov/users/cgamboa/testDMZ_new_2 for write
04 Apr 2022 14:57:53 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] OPAQUE : {tried=dc212.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc052.usatlas.bnl.gov,dc201.usatlas.bnl.gov,dc221.usatlas.bnl.gov,dc209.usatlas.bnl.gov,dc226.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc216.usatlas.bnl.gov,dc213.usatlas.bnl.gov,dc215.usatlas.bnl.gov,dc220.usatlas.bnl.gov,dc217.usatlas.bnl.gov,dc048.usatlas.bnl.gov,dc212.usatlas.bnl.gov,dc226.usatlas.bnl.gov, org.dcache.xrootd.client=cgamboa.14241@lxplus760.cern.ch, org.dcache.uuid=e5ab9b05-6d39-43ea-9742-ed94e49fad19, oss.asize=5, triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr}
04 Apr 2022 14:57:53 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Redirecting to dc216.usatlas.bnl.gov, dc216.usatlas.bnl.gov/10.42.38.99:33309
04 Apr 2022 14:57:53 [pool-7-thread-87464] [door:Xrootd-dcdoor07@xrootd-dcdoor07Domain:AAXb2LQ7mag] Sending the following host information to the client: dc216.usatlas.bnl.gov

My account pool manager configuration

psu create unit -store cgamboa:USERS@osm
psu set storage unit cgamboa:USERS@osm -required=1
psu addto ugroup token-DISKONLYSTORE cgamboa:USERS@osm
psu create ugroup token-cgamboa
psu addto ugroup token-cgamboa cgamboa:USERS@osm
psu create link lnk-USER_cgamboa-readwrite any-protocol token-cgamboa world-net
psu set link lnk-USER_cgamboa-readwrite -readpref=80 -writepref=20 -cachepref=0 -p2ppref=0 -section=costpreference-DISKONLY
psu addto link lnk-USER_cgamboa-readwrite DDNDATA-replica
psu addto link lnk-USER_cgamboa-readwrite DDNDATA-replica2
psu addto link lnk-USER_cgamboa-readwrite DDNDATA-replica3
psu addto link lnk-USER_cgamboa-readwrite MCDISK-readwrite
psu addto link lnk-USER_cgamboa-readwrite PRIMEDISKONLY
psu addto link lnk-USER_cgamboa-readwrite SECONDARYDISKONLY
psu create link lnk-dmz-USER_cgamboa-read remote-transfer-protocol token-cgamboa world-net
psu set link lnk-dmz-USER_cgamboa-read -readpref=99 -writepref=0 -cachepref=0 -p2ppref=99 -section=costpreference-dmz
psu addto link lnk-dmz-USER_cgamboa-read dmz-pools
psu create link lnk-dmz-USER_cgamboa-write **remote-transfer-protocol** token-cgamboa world-net
psu set link lnk-dmz-USER_cgamboa-write -readpref=0 -writepref=99 -cachepref=0 -p2ppref=99 -section=costpreference-dmz
psu addto link lnk-dmz-USER_cgamboa-write dmz-pools
psu create linkGroup user-cgamboa-link-group
psu set linkGroup custodialAllowed user-cgamboa-link-group false
psu set linkGroup replicaAllowed user-cgamboa-link-group true
psu set linkGroup nearlineAllowed user-cgamboa-link-group true
psu set linkGroup outputAllowed user-cgamboa-link-group false
psu set linkGroup onlineAllowed user-cgamboa-link-group true
psu addto linkGroup user-cgamboa-link-group lnk-USER_cgamboa-readwrite
psu create link lnk-dmz-pools any-store **remote-transfer-protocol** world-net
psu set link lnk-dmz-pools -readpref=99 -writepref=99 -cachepref=0 -p2ppref=99 -section=costpreference-dmz
psu addto link lnk-dmz-pools dmz-pools
psu create ugroup remote-transfer-protocol
psu addto ugroup remote-transfer-protocol RemoteHttpsDataTransfer/1
psu addto ugroup remote-transfer-protocol xrootd/*

I can also add the entire PoolManager if you wish but it has 9.5k lines.

We could also schedule a call if this ease the process

All the best, Carlos

cfgamboa commented 2 years ago

The DMZ pools naming convention is dcdoor*_1 The dCache configuration for the door is below: ....

[xrootd-${host.name}Domain]
[xrootd-${host.name}Domain/xrootd]
xrootd.cell.name=Xrootd-${host.name}
xrootd.net.port=1096

[xrootd2-${host.name}Domain]
[xrootd2-${host.name}Domain/xrootd]
xrootd.cell.name=Xrootd2-${host.name}
xrootd.net.port=1097

[webdav2-${host.name}_httpsDomain/webdav]
webdav.cell.name=WebDAV2-${host.name}-2
webdav.redirect.on-read=false
webdav.redirect.on-write=false
webdav.authz.anonymous-operations=NONE
webdav.authn.protocol=https
webdav.authn.accept-client-cert=true
webdav.authn.require-client-cert=false
webdav.service.gplazma.timeout=180000
webdav.enable.overwrite=true
webdav.net.port=443
webdav.limits.acceptors=3
webdav.limits.idle-time=900
webdav.limits.idle-time.unit=SECONDS
webdav.limits.low-resource-idle-time=900
webdav.limits.low-resource-idle-time.unit=SECONDS
webdav.limits.backlog=2048
webdav.limits.threads.max=1500
webdav.limits.threads.min=2
webdav.limits.queue-length=1500
webdav.mover.timeout=28800000
webdav.net.internal=10.42.38.62
alrossi commented 2 years ago

So, just to make sure I get the full picture (including pool groups) could you please attach (don't insert) the full poolmanager.conf?

Also, could you do:

\s PoolManager pm ls -l

and include the output?

Thanks.

cfgamboa commented 2 years ago

Hi @alrossi

I will send the file via the dcache-rt ticket, [10308].

[dcadmin02] (xrootd-dcdoor07@xrootd-dcdoor07Domain) admin > \c PoolManager 
[dcadmin02] (PoolManager@dCacheDomain) admin > \s PoolManager pm ls -l
Inherited by all partitions
costpreference-RESILIENT (wass)
    -alert=1.0
    -cpucostfactor=1.0
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.1
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=yes
    -slope=0.0
    -spacecostfactor=0.0
    -stage-allowed=no
costpreference-TAPEWRITE (wass)
    -alert=0.0
    -cpucostfactor=0.0
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=yes
    -slope=0.0
    -spacecostfactor=0.0
    -stage-allowed=no
costpreference-HOPPING (wass)
    -alert=1.0
    -cpucostfactor=1.0
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.2
    -p2p-allowed=yes
    -p2p-fortransfer=yes
    -p2p-oncost=yes
    -slope=0.0
    -spacecostfactor=1.0
    -stage-allowed=no
default (wass)
    -alert=0.0
    -cpucostfactor=1.0
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.0
    -p2p-allowed=yes
    -sameHostCopy=besteffort
    -sameHostRetry=notchecked
    -slope=0.0
    -spacecostfactor=1.0
    -stage-allowed=yes
    -stage-oncost=no
costpreference-HPCDISK (wass)
    -alert=0.0
    -cpucostfactor=0.999
    -fallback=0.02
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=no
    -slope=0.0
    -spacecostfactor=0.0010
    -stage-allowed=no
costpreference-HPCTAPEREADWRITE (wass)
    -alert=0.0
    -cpucostfactor=0.999
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=no
    -slope=0.0
    -spacecostfactor=0.00000010
    -stage-allowed=yes
    -stage-oncost=no
costpreference-DISKONLY (wass)
    -alert=0.0
    -cpucostfactor=0.999
    -fallback=0.02
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=no
    -slope=0.0
    -spacecostfactor=0.0010
    -stage-allowed=no
costpreference-dmz (wass)
    -alert=1.0
    -cpucostfactor=1.0
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=6
    -p2p=0.1
    -p2p-allowed=yes
    -p2p-fortransfer=yes
    -p2p-oncost=yes
    -slope=0.0
    -spacecostfactor=0.1
    -stage-allowed=no
costpreference-TAPEREAD (wass)
    -alert=0.0
    -cpucostfactor=1
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=no
    -slope=0.0
    -spacecostfactor=1
    -stage-allowed=yes
    -stage-oncost=no
costpreference-TAPEWRITESTAGE (wass)
    -alert=0.0
    -cpucostfactor=1
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=yes
    -slope=0.0
    -spacecostfactor=1
    -stage-allowed=yes
    -stage-oncost=no
costpreference-MCDATATAPESTAGE (wass)
    -alert=0.0
    -cpucostfactor=1
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=no
    -slope=0.0
    -spacecostfactor=1
    -stage-allowed=yes
    -stage-oncost=no
costpreference-LAKELOCALGROUPDISK-STAGE (wass)
    -alert=0.0
    -cpucostfactor=1
    -fallback=0.0
    -halt=0.0
    -idle=0.0
    -max-copies=2
    -p2p=0.8
    -p2p-allowed=yes
    -p2p-fortransfer=no
    -p2p-oncost=no
    -slope=0.0
    -spacecostfactor=1
    -stage-allowed=yes
    -stage-oncost=no
alrossi commented 2 years ago

Thank you! I'll have a look and get back to you soon (probably tomorrow at this point ...)

cfgamboa commented 2 years ago

I appreciate your feedback. If needed we could set a Zoom meeting tomorrow.

All the best, Carlos

alrossi commented 2 years ago

Carlos, I have been looking at the config file, but before I go any further, could you perhaps clarify the following statement?

Please note that internal transfers using XROOT door do not have this issue.

Unfortunately the DMZ pool is not used by the Xroot door. But even in that situation the internal pool will need to be told to use the internal interface.

What you are saying in the first part is that the transfer succeeds if it is from the internal/worker node, but that the DMZ pools are still not used, correct?

I am not sure what you mean, however, by "the internal pool will need to be told to use the internal interface." What happens in dCache is that a pool is matched up with a client. If the client is given the pool endpoint to connect to, but because of firewall rules it really can't read or write to it, then we have a situation like what you are experiencing. This is what I expect is the case.

I have a working idea of why, on the basis of your configuration, xrootd is not getting the DMZ pools ... , but I want to make sure I am not misunderstanding something first.

cfgamboa commented 2 years ago

Hello Al

  1. Transfers from clients on the BNL workers nodes can write to dCache via Xrootd. Internally the worker nodes have internal network connectivity, no DMZ is involved.
  2. As it is transfers from external clients could not write to dCache via Xrootd 2.1 Internal pools can not access external clients IPs. (this is by designed the pools are isolated) 2.2 DMZ is meant to be used to allow DMZ pools interact with the internal storage for request that come from external clients 2.3 The tests done with the DMZ enabled for XROOT protocol does not appear to work. Still the transfer attempts to connect to the internal pool instead of the DMZ pool.

As you have proposed via RT we will continue this there.

All the best, Carlos

alrossi commented 2 years ago

Problem solved via configuration as suggested.

alrossi commented 1 year ago

Hi Carlos,

Sorry, I'm not following here. Is there still a problem with this? I thought that we decided that for BNL fixing xrootd via poolmanager.conf was too complicated and precarious, and that you are using the proxy.

Could you perhaps elaborate on your question here?

Thanks, Al

Albert L. Rossi Senior Software Developer Scientific Computing Division, Scientific Data Services, Distributed Data Development WH 566 Fermi National Accelerator Laboratory Batavia, IL 60510 (630) 840-3023

From: gamboa @.> Sent: Tuesday, October 11, 2022 2:36 AM To: dCache/dcache @.> Cc: Albert Rossi @.>; State change @.> Subject: Re: [dCache/dcache] (General problem: Unable to find address that faces lxplus727.cern.ch/2001:1458:d00:14:0:0:0:63 [666]) (Issue #6568)

Hello Al,

I am using my user hat to test, but I am also “recently" helping in supporting the dCache infrastructure at BNL. I looked briefly at the configuration of the pool manager and even attempted to run a simple test using different reference like this one (https://github.com/dCache/dcache/blob/6a2bc7a0ff3a9d14fb6a381b0643c9b1c03219f0/docs/TheBook/src/main/markdown/config-PoolManager.md#the-pool-selection-mechanismhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_blob_6a2bc7a0ff3a9d14fb6a381b0643c9b1c03219f0_docs_TheBook_src_main_markdown_config-2DPoolManager.md-23the-2Dpool-2Dselection-2Dmechanism&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=5mVxXKiPi7ug3BI4juyybdWC5-WW4kqjQ4q6qf-_c3Ijb9x5KfQjgZpK30zieYLf&s=w0O6Di-GFlau8KB9Oitx44o36nDpzE4GsSR8l5f0KTM&e= https://github.com/dCache/dcache/blob/6a2bc7a0ff3a9d14fb6a381b0643c9b1c03219f0/docs/TheBook/src/main/markdown/config-PoolManager.md#the-pool-selection-mechanismhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_blob_6a2bc7a0ff3a9d14fb6a381b0643c9b1c03219f0_docs_TheBook_src_main_markdown_config-2DPoolManager.md-23the-2Dpool-2Dselection-2Dmechanism-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=5mVxXKiPi7ug3BI4juyybdWC5-WW4kqjQ4q6qf-_c3Ijb9x5KfQjgZpK30zieYLf&s=LnUwbzUg-KoHaD8p-GQTHKKV-A36OYufiI5rIVHBGGg&e=)

Please note that internal transfers using XROOT door do not have this issue.

Unfortunately the DMZ pool is not used by the Xroot door. But even in that situation the internal pool will need to be told to use the internal interface.

I assume that you did not observed something unusual on the logs.

Thank you for advising,

Carlos

On Apr 1, 2022, at 2:47 PM, Albert Rossi @.***> wrote:

Hi Carlos,

Well, xrootd does not those adapters available to it (at least as far as I know). To get xrootd to work, you need to rely on the pool setup, as I said above.

I can't say much more about this without input from BNL. I suggest speaking with them. There must be other xrootd users. Are they having similar issues?

Cheers, Al

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6568#issuecomment-1086222478https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_6568-23issuecomment-2D1086222478-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=5mVxXKiPi7ug3BI4juyybdWC5-WW4kqjQ4q6qf-_c3Ijb9x5KfQjgZpK30zieYLf&s=Q2-4lmnCrjVgVpEMxcWWfNYXyY-qFvg9liOys6Vq0kA&e=, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO46FDALHM7HRDZ2HYDVC5AEJANCNFSM5SAUAKRQhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AHIHMO46FDALHM7HRDZ2HYDVC5AEJANCNFSM5SAUAKRQ-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=5mVxXKiPi7ug3BI4juyybdWC5-WW4kqjQ4q6qf-_c3Ijb9x5KfQjgZpK30zieYLf&s=EvEhS5MgcmiuWFH3EByikwnbnXap5G2-4HWdoaS5NyU&e=. You are receiving this because you authored the thread.

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_6568-23issuecomment-2D1274224167&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=5mVxXKiPi7ug3BI4juyybdWC5-WW4kqjQ4q6qf-_c3Ijb9x5KfQjgZpK30zieYLf&s=DsVLOhxSdIKzyAHInInVVeLPapEQtLzD13fzzVvmpvk&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA6NBHGYLSVDNST7F2QO4OTWCUKHRANCNFSM5SAUAKRQ&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=5mVxXKiPi7ug3BI4juyybdWC5-WW4kqjQ4q6qf-_c3Ijb9x5KfQjgZpK30zieYLf&s=patz2luZTQzx9H1gkFqvhqbOrpeNR945z2t-1uQbObA&e=. You are receiving this because you modified the open/close state.Message ID: @.***>

cfgamboa commented 1 year ago

Hello Al,

This is an old closed ticket. No need to follow up. Sorry for the noise.

Carlos

alrossi commented 1 year ago

Hi Carlos.

OK, thanks!

Al

Albert L. Rossi Senior Software Developer Scientific Computing Division, Scientific Data Services, Distributed Data Development WH 566 Fermi National Accelerator Laboratory Batavia, IL 60510 (630) 840-3023

From: gamboa @.> Sent: Tuesday, October 11, 2022 9:37 AM To: dCache/dcache @.> Cc: Albert Rossi @.>; State change @.> Subject: Re: [dCache/dcache] (General problem: Unable to find address that faces lxplus727.cern.ch/2001:1458:d00:14:0:0:0:63 [666]) (Issue #6568)

Hello Al,

This is an old closed ticket. No need to follow up. Sorry for the noise.

Carlos

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_6568-23issuecomment-2D1274800634&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=WEeXV9VZb55T3A7FuybYQi2U3Z9i34h_I6OlNMuo4HTmWQlCQ9MhmLGdkZU6u9IO&s=LnkMbmPN9t2J2aUQpHhxc5sqHQCJ7uOe4cCjA09KOW4&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA6NBHGLVZUJZHRAUE5S4PLWCV3R5ANCNFSM5SAUAKRQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=WEeXV9VZb55T3A7FuybYQi2U3Z9i34h_I6OlNMuo4HTmWQlCQ9MhmLGdkZU6u9IO&s=H-52VqWj_YOBViCVsExnCrYJWi1OE3fwpbN7VZYRQo4&e=. You are receiving this because you modified the open/close state.Message ID: @.***>