search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
277 stars 132 forks source link

door: java.net.UnknownHostException: DNS name not found [response code 3] #7535

Closed cfgamboa closed 3 months ago

cfgamboa commented 3 months ago

Hello,

This transfer is failing with error DNS name not found [response code 3], this is not persistent in some cases the same transfer succeeded.

The resolution of the doors on the srm is working as usual.

The dcache version is dcache-8.2.26-1.noarch

Could you please advise.

All the best, Carlos

Bellow the entire trace

 gfal-copy -f -vvv davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
DEBUG     no GFAL_CONFIG_DIR env var found, try to load configuration from default directory /etc/gfal2.d/
DEBUG     try to load configuration file /etc/gfal2.d//bdii.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//gfal2_core.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//x509.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//dcap_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//http_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//sftp_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//gsiftp_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//srm_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//xrootd_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//rfio_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//lfc_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//mock_plugin.conf ...
DEBUG    Using credentials from X509_USER_PROXY
DEBUG    Certificate: /tmp/x509up_h8539406517407983927_1a964fd4019e0179
DEBUG    Private key: /tmp/x509up_h8539406517407983927_1a964fd4019e0179
DEBUG    ... no GFAL_PLUGIN_DIR environment variable specified, try to load plugins in the default directory : /usr/lib64/gfal2-plugins/
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_http.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_file.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_mock.so 
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so
DEBUG    Davix: Enable GRID profile for DAVIX
DEBUG    Davix: Adding CA path /etc/grid-security/certificates/ to valid CA path list
DEBUG    Davix: Using X509_USER_PROXY to supply credentials: /tmp/x509up_h8539406517407983927_1a964fd4019e0179
DEBUG    Davix: HTTP/SSL Session caching ENABLED
DEBUG    Davix: Redirection Session caching ENABLED
DEBUG    Davix: libdavix path /lib64/libdavix.so.0, version: 0.8.4
DEBUG    Davix: Enable GRID profile for DAVIX
DEBUG    Davix: Adding CA path /etc/grid-security/certificates/ to valid CA path list
DEBUG    Davix: Using X509_USER_PROXY to supply credentials: /tmp/x509up_h8539406517407983927_1a964fd4019e0179
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_http.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_file.so
DEBUG     -> [gridftp_plugin] try to load ..
DEBUG     define GSIFTP session re-use to TRUE
DEBUG     -> [gridftp_plugin] loaded ..
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so
DEBUG    lib rfio is not defined in LCG_RFIO_TYPE, try to found it 
DEBUG    rfio library libdpm.so.1 found! configured to use it
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_mock.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_mock.so
DEBUG     plugin priority order: lfc-2.21.5 -> dcap-2.21.5 -> http-2.21.5 -> sftp-2.21.5 -> file-2.21.5 -> gridftp-2.21.5 -> srm-2.21.5 -> xrootd-2.21.5 -> rfio-2.21.5 -> mock-2.21.5 -> 
DEBUG    Service endpoint resolution, resolved from FULL SURL srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 -> httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
DEBUG    SRM context not available
DEBUG     SRM connection keep-alive 1
DEBUG    Impossible to get integer parameter SRM PLUGIN:OPERATION_TIMEOUT, set to default value 1800, err Key file does not have key “OPERATION_TIMEOUT” in group “SRM PLUGIN”
DEBUG     SRM operation timeout 1800
DEBUG     SRM connection timeout 60
DEBUG     SRM using certificate /tmp/x509up_h8539406517407983927_1a964fd4019e0179
DEBUG     SRM using private key /tmp/x509up_h8539406517407983927_1a964fd4019e0179
DEBUG       [gfal_srm_statG] try to stat file srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
DEBUG       [gfal_srm_statG] store srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 stat info in cache
DEBUG    Impossible to get string_list parameter DAV:DCBLDOOR03.SDCC.BNL.GOV:HEADERS, set to a default value (null), err Key file does not have group “DAV:DCBLDOOR03.SDCC.BNL.GOV”
DEBUG    Impossible to get string_list parameter HTTP PLUGIN:HEADERS, set to a default value (null), err Key file does not have key “HEADERS” in group “HTTP PLUGIN”
DEBUG    Impossible to get integer parameter HTTP PLUGIN:OPERATION_TIMEOUT, set to default value 1800, err Key file does not have key “OPERATION_TIMEOUT” in group “HTTP PLUGIN”
DEBUG    Using client X509 for HTTPS session authorization
DEBUG    Impossible to get string parameter BEARER:TOKEN, set to default value (null), err Key file does not have group “BEARER”
DEBUG    Impossible to get string_list parameter DAV:DCBLDOOR03.SDCC.BNL.GOV:HEADERS, set to a default value (null), err Key file does not have group “DAV:DCBLDOOR03.SDCC.BNL.GOV”
DEBUG    Impossible to get string_list parameter HTTP PLUGIN:HEADERS, set to a default value (null), err Key file does not have key “HEADERS” in group “HTTP PLUGIN”
DEBUG    Impossible to get integer parameter HTTP PLUGIN:OPERATION_TIMEOUT, set to default value 1800, err Key file does not have key “OPERATION_TIMEOUT” in group “HTTP PLUGIN”
DEBUG    Using client X509 for HTTPS session authorization
DEBUG    Davix: Create HttpRequest for https://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root
DEBUG    Davix:  -> negotiateRequest
DEBUG    Davix: NEON start internal request
DEBUG    Davix: no cached ne_session, create a new one 
DEBUG    Davix: HTTP session to https://dcbldoor03.sdcc.bnl.gov:443 begins.
DEBUG    Davix: configure session...
DEBUG    Davix: define connection timeout to 30
DEBUG    Davix: define operation timeout to 1800
DEBUG    Davix: add CA PATH /etc/grid-security/certificates/
DEBUG    Davix: disable login/password authentication
DEBUG    Davix: enable client cert authentication by callback 
DEBUG    Davix: Running pre_send hooks
INFO     Davix: > POST /pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root HTTP/1.1
> User-Agent: gfal2-util/1.8.0 gfal2/2.21.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: dcbldoor03.sdcc.bnl.gov
> Content-Type: application/macaroon-request
> Content-Length: 61
> 

DEBUG    Davix: Sending request-line and headers:
DEBUG    Davix: Doing DNS lookup on dcbldoor03.sdcc.bnl.gov...
DEBUG    Davix: clicert callback 
DEBUG    Davix: call client cert callback 
DEBUG    Davix: Sending request body:
DEBUG    Davix: Request body sent successfully
DEBUG    Davix: Request sent; retry is 0.
INFO     Davix: < HTTP/1.1 200 OK
INFO     Davix: < Date: Fri, 22 Mar 2024 15:42:04 GMT
INFO     Davix: < X-Content-Type-Options: nosniff
INFO     Davix: < X-Frame-Options: SAMEORIGIN
INFO     Davix: < X-XSS-Protection: 1; mode=block
INFO     Davix: < Server: dCache/8.2.26
INFO     Davix: < Content-Security-Policy: default-src 'none' ; img-src 'self' data: ; style-src 'self' 'unsafe-inline' ; script-src 'self'; font-src 'self'
INFO     Davix: < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
INFO     Davix: < Referrer-Policy: strict-origin-when-cross-origin
INFO     Davix: < Content-Type: application/json
INFO     Davix: < Content-Length: 2412
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_headers hooks
DEBUG    Davix:  <- negotiateRequest
DEBUG    Davix: Davix::BackendRequest::readSegment: want to read 1048576 bytes 
DEBUG    Davix: Reading 2412 bytes of response body.

DEBUG    Davix: Got 2412 bytes.
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 2412 bytes
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 0 bytes
DEBUG    Davix: Destroy HttpRequest
DEBUG    Davix: Running post_send hooks
DEBUG    Davix: Running destroy hooks.
DEBUG    Davix: Request ends.
DEBUG    Davix: add old session to cache httpsdcbldoor03.sdcc.bnl.gov:443
DEBUG    (SEToken) Set bearer token in credential_map[davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root] (access=read) (validity=180)
INFO     Using bearer token for HTTPS request authorization
DEBUG    Davix: Create HttpRequest for davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root
DEBUG    Davix:  -> executeRequest
DEBUG    Davix:  -> negotiateRequest
DEBUG    Davix: NEON start internal request
DEBUG    Davix: no cached ne_session, create a new one 
DEBUG    Davix: HTTP session to https://dcbldoor03.sdcc.bnl.gov:443 begins.
DEBUG    Davix: configure session...
DEBUG    Davix: define connection timeout to 30
DEBUG    Davix: define operation timeout to 1800
DEBUG    Davix: add CA PATH /etc/grid-security/certificates/
DEBUG    Davix: disable login/password authentication
DEBUG    Davix: enable client cert authentication by callback 
DEBUG    Davix: Running pre_send hooks
INFO     Davix: > PROPFIND /pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root HTTP/1.1
> User-Agent: gfal2-util/1.8.0 gfal2/2.21.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: dcbldoor03.sdcc.bnl.gov
> Depth: 0
> Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> 

DEBUG    Davix: Sending request-line and headers:
DEBUG    Davix: Doing DNS lookup on dcbldoor03.sdcc.bnl.gov...
DEBUG    Davix: clicert callback 
DEBUG    Davix: call client cert callback 
DEBUG    Davix: Request sent; retry is 0.
INFO     Davix: < HTTP/1.1 207 Multi-status
INFO     Davix: < Date: Fri, 22 Mar 2024 15:42:04 GMT
INFO     Davix: < X-Content-Type-Options: nosniff
INFO     Davix: < X-Frame-Options: SAMEORIGIN
INFO     Davix: < X-XSS-Protection: 1; mode=block
INFO     Davix: < Server: dCache/8.2.26
INFO     Davix: < Content-Security-Policy: default-src 'none' ; img-src 'self' data: ; style-src 'self' 'unsafe-inline' ; script-src 'self'; font-src 'self'
INFO     Davix: < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
INFO     Davix: < Referrer-Policy: strict-origin-when-cross-origin
INFO     Davix: < Content-Type: application/xml; charset=utf-8
INFO     Davix: < DAV: 1, 2
INFO     Davix: < Transfer-Encoding: chunked
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_headers hooks
DEBUG    Davix:  <- negotiateRequest
DEBUG    Davix: Bad server answer: Content-Length Invalid, impossible to determine answer size
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: [chunk] < 39C

DEBUG    Davix: Got chunk size: 924
DEBUG    Davix: Reading 924 bytes of response body.

DEBUG    Davix: Got 924 bytes.
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 924 bytes
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: [chunk] < 0

DEBUG    Davix: Got chunk size: 0
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 0 bytes
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_send hooks
DEBUG    Davix:  <- executeRequest
DEBUG    Davix: XML: Parsing 925 bytes.
DEBUG    Davix: XML: start-element (0, {DAV:, multistatus}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, response}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, href}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  href/filename parsed -> physics.0030.02279.HLT06.m01.f00000.root 
DEBUG    Davix: XML: end-element (1, {DAV:, href})
DEBUG    Davix:  properties detected 
DEBUG    Davix: XML: start-element (1, {DAV:, propstat}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, prop}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, getcontentlength}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  content length found -> parse it
DEBUG    Davix:  content length found -> 4938708525
DEBUG    Davix: XML: end-element (1, {DAV:, getcontentlength})
DEBUG    Davix: XML: start-element (1, {DAV:, getetag}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: XML: end-element (1, {DAV:, getetag})
DEBUG    Davix: XML: start-element (1, {DAV:, displayname}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: XML: end-element (1, {DAV:, displayname})
DEBUG    Davix: XML: start-element (1, {DAV:, getcontenttype}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: XML: end-element (1, {DAV:, getcontenttype})
DEBUG    Davix: XML: start-element (1, {DAV:, getlastmodified}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  getlastmodified found -> parse it 
DEBUG    Davix:  getlastmodified found -> value 1710801820 
DEBUG    Davix: XML: end-element (1, {DAV:, getlastmodified})
DEBUG    Davix: XML: start-element (1, {DAV:, creationdate}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: creationdate found -> parse it
DEBUG    Davix:  creationdate found -> value 1710801641 
DEBUG    Davix: XML: end-element (1, {DAV:, creationdate})
DEBUG    Davix: XML: end-element (1, {DAV:, prop})
DEBUG    Davix: XML: start-element (1, {DAV:, status}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  status found -> parse it
DEBUG    Davix:  status value : 200
DEBUG    Davix: XML: end-element (1, {DAV:, status})
DEBUG    Davix:  end of properties... 
DEBUG    Davix: XML: end-element (1, {DAV:, propstat})
DEBUG    Davix:  properties detected 
DEBUG    Davix: XML: start-element (1, {DAV:, propstat}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, prop}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, resourcetype}) => 1
DEBUG    Davix: XML: end-element (1, {DAV:, resourcetype})
DEBUG    Davix: XML: end-element (1, {DAV:, prop})
DEBUG    Davix: XML: start-element (1, {DAV:, status}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  status found -> parse it
DEBUG    Davix:  status value : 404
DEBUG    Davix: XML: end-element (1, {DAV:, status})
DEBUG    Davix:  end of properties... 
DEBUG    Davix: Bad status code ! properties dropped
DEBUG    Davix: XML: end-element (1, {DAV:, propstat})
DEBUG    Davix: XML: end-element (1, {DAV:, response})
DEBUG    Davix: XML: end-element (1, {DAV:, multistatus})
DEBUG    Davix: XML: xmlParseChunk returned 0
DEBUG    Davix: Destroy HttpRequest
DEBUG    Davix: Running destroy hooks.
DEBUG    Davix: Request ends.
DEBUG    Davix: add old session to cache httpsdcbldoor03.sdcc.bnl.gov:443
DEBUG    Davix:  davix_stat <-
DEBUG     srm_statG -> value taken from the cache
Copying 4938708525 bytes davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root => srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
DEBUG     -> Gfal::Transfer::FileCopy
event: [1711122124297] BOTH   GFAL2:CORE:COPY   LIST:ENTER  
INFO     Event triggered: BOTH GFAL2:CORE:COPY LIST:ENTER 
event: [1711122124297] BOTH   GFAL2:CORE:COPY   LIST:ITEM   davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root => srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
INFO     Event triggered: BOTH GFAL2:CORE:COPY LIST:ITEM davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root => srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
event: [1711122124297] BOTH   GFAL2:CORE:COPY   LIST:EXIT   
INFO     Event triggered: BOTH GFAL2:CORE:COPY LIST:EXIT 
DEBUG    Endpoint not SRM: davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root
DEBUG    Service endpoint resolution, resolved from FULL SURL srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 -> httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
DEBUG    SRM context recycled for httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
INFO     Endpoint of type dCache: srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
DEBUG    No Castor endpoint. Honor configuration for SESSION_REUSE
event: [1711122124302] BOTH   SRM   PREPARE:ENTER   
INFO     Event triggered: BOTH SRM PREPARE:ENTER 
DEBUG           Checksum check enabled: 0
DEBUG    Impossible to get string_list parameter DAV:DCBLDOOR03.SDCC.BNL.GOV:HEADERS, set to a default value (null), err Key file does not have group “DAV:DCBLDOOR03.SDCC.BNL.GOV”
DEBUG    Impossible to get string_list parameter HTTP PLUGIN:HEADERS, set to a default value (null), err Key file does not have key “HEADERS” in group “HTTP PLUGIN”
DEBUG    Impossible to get integer parameter HTTP PLUGIN:OPERATION_TIMEOUT, set to default value 1800, err Key file does not have key “OPERATION_TIMEOUT” in group “HTTP PLUGIN”
DEBUG    Using client X509 for HTTPS session authorization
DEBUG    (SEToken) Found token in credential_map[davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root] (access=read) (needed=read)
INFO     Using bearer token for HTTPS request authorization
DEBUG    Davix: Create HttpRequest for davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root
DEBUG    Davix:  -> executeRequest
DEBUG    Davix:  -> negotiateRequest
DEBUG    Davix: NEON start internal request
DEBUG    Davix: cached ne_session found ! taken from cache 
DEBUG    Davix: configure session...
DEBUG    Davix: define connection timeout to 30
DEBUG    Davix: define operation timeout to 1800
DEBUG    Davix: add CA PATH /etc/grid-security/certificates/
DEBUG    Davix: disable login/password authentication
DEBUG    Davix: enable client cert authentication by callback 
DEBUG    Davix: Running pre_send hooks
INFO     Davix: > PROPFIND /pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root HTTP/1.1
> User-Agent: gfal2-util/1.8.0 gfal2/2.21.5 neon/0.0.29
> TE: trailers
> Host: dcbldoor03.sdcc.bnl.gov
> Depth: 0
> Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> 

DEBUG    Davix: Sending request-line and headers:
DEBUG    Davix: Request sent; retry is 1.
INFO     Davix: < HTTP/1.1 207 Multi-status
INFO     Davix: < Date: Fri, 22 Mar 2024 15:42:04 GMT
INFO     Davix: < X-Content-Type-Options: nosniff
INFO     Davix: < X-Frame-Options: SAMEORIGIN
INFO     Davix: < X-XSS-Protection: 1; mode=block
INFO     Davix: < Server: dCache/8.2.26
INFO     Davix: < Content-Security-Policy: default-src 'none' ; img-src 'self' data: ; style-src 'self' 'unsafe-inline' ; script-src 'self'; font-src 'self'
INFO     Davix: < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
INFO     Davix: < Referrer-Policy: strict-origin-when-cross-origin
INFO     Davix: < Content-Type: application/xml; charset=utf-8
INFO     Davix: < DAV: 1, 2
INFO     Davix: < Transfer-Encoding: chunked
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_headers hooks
DEBUG    Davix:  <- negotiateRequest
DEBUG    Davix: Bad server answer: Content-Length Invalid, impossible to determine answer size
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: [chunk] < 39C

DEBUG    Davix: Got chunk size: 924
DEBUG    Davix: Reading 924 bytes of response body.

DEBUG    Davix: Got 924 bytes.
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 924 bytes
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: [chunk] < 0

DEBUG    Davix: Got chunk size: 0
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 0 bytes
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_send hooks
DEBUG    Davix:  <- executeRequest
DEBUG    Davix: XML: Parsing 925 bytes.
DEBUG    Davix: XML: start-element (0, {DAV:, multistatus}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, response}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, href}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  href/filename parsed -> physics.0030.02279.HLT06.m01.f00000.root 
DEBUG    Davix: XML: end-element (1, {DAV:, href})
DEBUG    Davix:  properties detected 
DEBUG    Davix: XML: start-element (1, {DAV:, propstat}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, prop}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, getcontentlength}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  content length found -> parse it
DEBUG    Davix:  content length found -> 4938708525
DEBUG    Davix: XML: end-element (1, {DAV:, getcontentlength})
DEBUG    Davix: XML: start-element (1, {DAV:, getetag}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: XML: end-element (1, {DAV:, getetag})
DEBUG    Davix: XML: start-element (1, {DAV:, displayname}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: XML: end-element (1, {DAV:, displayname})
DEBUG    Davix: XML: start-element (1, {DAV:, getcontenttype}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: XML: end-element (1, {DAV:, getcontenttype})
DEBUG    Davix: XML: start-element (1, {DAV:, getlastmodified}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  getlastmodified found -> parse it 
DEBUG    Davix:  getlastmodified found -> value 1710801820 
DEBUG    Davix: XML: end-element (1, {DAV:, getlastmodified})
DEBUG    Davix: XML: start-element (1, {DAV:, creationdate}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix: creationdate found -> parse it
DEBUG    Davix:  creationdate found -> value 1710801641 
DEBUG    Davix: XML: end-element (1, {DAV:, creationdate})
DEBUG    Davix: XML: end-element (1, {DAV:, prop})
DEBUG    Davix: XML: start-element (1, {DAV:, status}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  status found -> parse it
DEBUG    Davix:  status value : 200
DEBUG    Davix: XML: end-element (1, {DAV:, status})
DEBUG    Davix:  end of properties... 
DEBUG    Davix: XML: end-element (1, {DAV:, propstat})
DEBUG    Davix:  properties detected 
DEBUG    Davix: XML: start-element (1, {DAV:, propstat}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, prop}) => 1
DEBUG    Davix: XML: start-element (1, {DAV:, resourcetype}) => 1
DEBUG    Davix: XML: end-element (1, {DAV:, resourcetype})
DEBUG    Davix: XML: end-element (1, {DAV:, prop})
DEBUG    Davix: XML: start-element (1, {DAV:, status}) => 1
DEBUG    Davix: XML: char-data (1) returns 0
DEBUG    Davix:  status found -> parse it
DEBUG    Davix:  status value : 404
DEBUG    Davix: XML: end-element (1, {DAV:, status})
DEBUG    Davix:  end of properties... 
DEBUG    Davix: Bad status code ! properties dropped
DEBUG    Davix: XML: end-element (1, {DAV:, propstat})
DEBUG    Davix: XML: end-element (1, {DAV:, response})
DEBUG    Davix: XML: end-element (1, {DAV:, multistatus})
DEBUG    Davix: XML: xmlParseChunk returned 0
DEBUG    Davix: Destroy HttpRequest
DEBUG    Davix: Running destroy hooks.
DEBUG    Davix: Request ends.
DEBUG    Davix: add old session to cache httpsdcbldoor03.sdcc.bnl.gov:443
DEBUG    Davix:  davix_stat <-
DEBUG           No SRM resolution needed on davs://dcbldoor03.sdcc.bnl.gov:443/pnfs/sdcc.bnl.gov/data/bellediskdata/TMP/belle/Raw/e0030/physics/r02279/sub00/physics.0030.02279.HLT06.m01.f00000.root
DEBUG           PUT surl -> turl resolution start 
DEBUG    Trying to delete srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
DEBUG    Service endpoint resolution, resolved from FULL SURL srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 -> httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
DEBUG    SRM context recycled for httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
DEBUG    srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 deleted with success
event: [1711122124354] DEST   SRM   OVERWRITE   Deleted srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
INFO     Event triggered: DESTINATION SRM OVERWRITE Deleted srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10
DEBUG           Initial TURLs: gsiftp;https;root
DEBUG           Reordered TURLs: https;gsiftp;root
DEBUG    Service endpoint resolution, resolved from FULL SURL srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 -> httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
DEBUG    SRM context recycled for httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
INFO     Got PUT token for srm://dcblsrm.sdcc.bnl.gov/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10: e8ceb560:-378847862
WARNING  Transfer failed with: [srm_resolve_turls] DESTINATION SRM_PUT_TURL [srm_resolve_put_turl][gfal_srm_put_rd3_turl][gfal_srm_put_rd3_turl] error on the turl  request : [SE][PrepareToPut][SRM_INTERNAL_ERROR] Failed to resolve door: java.net.UnknownHostException: DNS name not found [response code 3] 
INFO     Rolling back PUT
DEBUG     <- Gfal::Transfer::FileCopy
gfal-copy error: 70 (Communication error on send) - [gfalt_copy_file][perform_copy][srm_plugin_filecopy][srm_resolve_turls] DESTINATION SRM_PUT_TURL [srm_resolve_put_turl][gfal_srm_put_rd3_turl][gfal_srm_put_rd3_turl] error on the turl  request : [SE][PrepareToPut][SRM_INTERNAL_ERROR] Failed to resolve door: java.net.UnknownHostException: DNS name not found [response code 3]
cfgamboa commented 3 months ago

From 


DEBUG           Initial TURLs: gsiftp;https;root
DEBUG           Reordered TURLs: https;gsiftp;root
DEBUG    Service endpoint resolution, resolved from FULL SURL srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape/belle/Raw/e0030/physics/r02279/sub00/test.door05.nc.10 -> httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2
DEBUG    SRM context recycled for httpg://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2

SRM uses HTTPS to instantiate the transfer. It seems that DAVs uses /etc/hosts to Resolve IP/Hostname. Only one door had the internal IPV6 hostname enabled.

The issue seems to be now cositently addressed by including all internal IPV6 component in the other 2 doors.

After that my test did not failed. I see similar FTS jobs successfully finishing in the last 2 hours.

https://fts.usatlas.bnl.gov:8449/fts3/ftsmon/#/errors/pairs?source_se=&dest_se=srm:%2F%2Fdcblsrm.sdcc.bnl.gov&time_window=2&vo= in the la