Closed StevenCTimm closed 4 months ago
Now fts3 works with xrdcp 3rd party transfer on this machine, will watch to make sure they get through the voms2 transition OK then configure the RSE.
we are through the voms transition and I have configured the RSE but the rules are still stuck investigating
now fts3 doesn't work anymore with xrdcp 3rd party transfer. will have to file a ticket with NERSC we can log into dtn14.nersc.gov and look at their xrootd configuration to see if they are doing anything wrong.
Filed INC0215872
What came out of it was that NERSC security has gone back on taking voms-authenticated proxies on the xrootd server and are going back to an individual certificate registration model as they always did on their gridftp servers. Have put some questions in on the incident, we will see what they say.
Pinged NERSC again re. our incident.
WE switched to doing 3rd party https.. turns out there are significant issues with 3rd party xrdcp when one of the ends of the transfer is based in dCache.. this is working now.
WE can do 3rd party xrdcp from dtn114.nersc.gov to fndca1.fnal.gov if I do it from the command line for example on dunesl7gpvm01. But the same 3rd party transfer initiated from fts3-public.fnal.gov, doesn't work we get this error sequence
INFO Mon, 20 Nov 2023 21:53:04 +0000; Configured to skip retrieval of SE-issued tokens. Retrieval delegated to downstream Gfal2 client INFO Mon, 20 Nov 2023 21:53:04 +0000; Transfer accepted INFO Mon, 20 Nov 2023 21:53:04 +0000; Proxy: /tmp/x509up_h6490419993171066944_5c8272978d824540 INFO Mon, 20 Nov 2023 21:53:04 +0000; VO: 5c8272978d824540 INFO Mon, 20 Nov 2023 21:53:04 +0000; Job id: 2e1c0c4c-87ef-11ee-8f30-0a580a670082 INFO Mon, 20 Nov 2023 21:53:04 +0000; File id: 6686963 INFO Mon, 20 Nov 2023 21:53:04 +0000; Source url: root://dtn114.nersc.gov:1094//global/cfs/cdirs/dunepro/RITM1893613/output/atmnu_max/50231162/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_0_20231118T173239Z_gen_g4_detsim_hitreco.root INFO Mon, 20 Nov 2023 21:53:04 +0000; Dest url: root://fndca1.fnal.gov//pnfs/fnal.gov/usr/dune/scratch/dunepro/ingest/FDHDVD_RITM1893613/HD/50231162/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_0_20231118T173239Z_gen_g4_detsim_hitreco.root INFO Mon, 20 Nov 2023 21:53:04 +0000; Overwrite enabled: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; Disable delegation: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; Disable local streaming: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; Evict source file: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; Dest space token: INFO Mon, 20 Nov 2023 21:53:04 +0000; Source space token: INFO Mon, 20 Nov 2023 21:53:04 +0000; Checksum: INFO Mon, 20 Nov 2023 21:53:04 +0000; Checksum enabled: No checksum comparison INFO Mon, 20 Nov 2023 21:53:04 +0000; User filesize: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; File metadata: INFO Mon, 20 Nov 2023 21:53:04 +0000; Job metadata: {\"auth_method\":?\"certificate\"} INFO Mon, 20 Nov 2023 21:53:04 +0000; Bringonline token: INFO Mon, 20 Nov 2023 21:53:04 +0000; UDT: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; BDII:lcg-bdii.cern.ch:2170 INFO Mon, 20 Nov 2023 21:53:04 +0000; Source token issuer: INFO Mon, 20 Nov 2023 21:53:04 +0000; Destination token issuer: INFO Mon, 20 Nov 2023 21:53:04 +0000; Report on the destination tape file: 0 INFO Mon, 20 Nov 2023 21:53:04 +0000; Third Party TURL protocol list: gsiftp;https;root INFO Mon, 20 Nov 2023 21:53:04 +0000; Getting source file size ERR Mon, 20 Nov 2023 21:53:08 +0000; Non recoverable error: [13] Failed to stat file (Permission denied)
Suspecting there's some subtle thing wrong in the auth config of their xrootd server such that it can't deal with a delegated proxy and/or that I somehow didn't delegate the proxy right.