search

DUNE / data-mgmt-ops

3 stars 2 forks source link

CTA backlog #652

Closed StevenCTimm closed 1 week ago

StevenCTimm commented 1 week ago

From fts3-public.cern.ch

root://eospublic.cern.ch root://eosctapublic.cern.ch dune 62671 64 - - - 1512 93 - 94.21 % 1562.31 MiB/s    


StevenCTimm commented 1 week ago

We've figured out that this big blurp of files heading to tape simultaneously came this morning when I put a bunch of pre-beam files into the declad dropbox, resulting in DUNE_CERN_EOS rules being made for some pretty big pre-beam datasets that didn't have them before. at one point the peak was 80K now it is closer to 60K so we are catching up.

https://cern.service-now.com/service-portal?id=ticket&is_new_order=true&table=incident&sys_id=796c1f10978b8a5081ef33f71153af3a (INC 3943610) has been filed with CERN service desk to ask for investigating the timeouts.

StevenCTimm commented 1 week ago

About half the backlog was cleared overnight, helped by the fact that we didn't take much data overnight. The timeouts have stopped. We still need to get a "bringonline" timeout into the FTS jobs we are sending to CTA.

StevenCTimm commented 1 week ago

All backlog is now cleared.