Right now, the status of a replication rule only provides a single high-level error summary. For example:
$ rucio rule-info --examine db3277cdd4324dcc87074cc8c84ee324
Status of the replication rule: TRANSFER_FAILED:TRANSFER [5] TRANSFER CHECKSUM MISMATCH USER_DEFINE and SRC checksums are different. 09e3c4ef != 8f14f45a
STUCK Requests:
cms:/store/mc/RunIISummer17PrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/MCv2_correctPU_94X_mc2017_realistic_v9-v1/20001/2E63DAAA-930C-E811-867F-0242AC130002.root
RSE: T3_US_NERSC_REAL
Attempts: 40
Last Retry: 2018-02-26 07:52:37
Last error: FAILED
Last source: T2_US_Nebraska_REAL
Available sources: T2_US_Nebraska_REAL
Blacklisted sources:
cms:/store/mc/RunIISummer17PrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/MCv2_correctPU_94X_mc2017_realistic_v9-v1/00032/2ABA00E8-A610-E811-ADDB-0242AC130002.root
RSE: T3_US_NERSC_REAL
Attempts: 40
Last Retry: 2018-02-26 13:38:41
Last error: FAILED
Last source: T2_US_Nebraska_REAL
Available sources: T2_US_Nebraska_REAL
Blacklisted sources:
(this particular case goes on for another 200 files or so...)
However, it's not practical to debug this rule because I have no clue what error corresponds to which file.
Modifications
Two suggestions:
In the output of rucio rule-info --examine ...., record the last error message per stuck file.
Also record the last FTS job ID where the file failed (will probably also need the FTS server URL).
Right now, the status of a replication rule only provides a single high-level error summary. For example:
(this particular case goes on for another 200 files or so...)
However, it's not practical to debug this rule because I have no clue what error corresponds to which file.
Modifications
Two suggestions:
rucio rule-info --examine ....
, record the last error message per stuck file.