search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
288 stars 136 forks source link

Bulk service: FileNotFoundCacheException Staging failures for existing files #7585

Closed vingar closed 3 months ago

vingar commented 4 months ago

Hello,

Some staging requests for the same files kept failing, with the following FTS error:

Error reason: STAGING [42] [Tape REST API] Missing response item for path=/pnfs/usatlas.bnl.gov/MCTAPE/atlasgrouptape/group/phys-higgs/mc16_13TeV/group.phys-higgs.mc16_13TeV.mc16a.Sherpa2_diphoton_myy_90_175.MxAODDetailed.e6452_s3126_r9364_p4097_h025.totape_20210406.root/mc16a.Sherpa2_diphoton_myy_90_175.MxAODDetailed.e6452_s3126_r9364_p4097_h025.008.totape_20210406.root

https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/e7eff772-1cd3-11ef-bbd2-fa163e52bca9

DDM dashboard: https://monit-grafana.cern.ch/goto/_73xvmsIR?orgId=17

Error in bulk tables:

error_type: diskCacheV111.util.FileNotFoundCacheException
error_message: CacheException(rc=10001;msg=No such file or directory /group/phys-higgs/mc16_13TeV/group.phys-higgs.mc16_13TeV.mc16a.Sherpa2_diphoton_myy_90_175.MxAODDetailed.e6452_s3126_r9364_p4097_h025.totape_20210406.root/mc16a.Sherpa2_diphoton_myy_90_175.MxAODDetailed

While it's possible to stage the file with the pnfsid as the file exists.

It maybe to be related to some limit on path length.

Here 299: /pnfs/usatlas.bnl.gov/MCTAPE/atlasgrouptape/group/phys-higgs/mc16_13TeV/group.phys-higgs.mc16_13TeV.mc16a.Sherpa2_diphoton_myy_90_175.MxAODDetailed.e6452_s3126_r9364_p4097_h025.totape_20210406.root/mc16a.Sherpa2_diphoton_myy_90_175.MxAODDetailed.e6452_s3126_r9364_p4097_h025.008.totape_20210406.root

DmitryLitvintsev commented 4 months ago

I believe this patch: https://rb.dcache.org/r/14255/ addressed the problem. It will be included in the next round of bug fix releases. (9.x and 10.x)