search

piperov / Rucio_storage_consistency

0 stars 2 forks source link

Failure due to not having the algorithm #2

Open ericvaandering opened 3 years ago

ericvaandering commented 3 years ago

When I run your code I get the following

 Processing dark file:
 RSE:  T2_US_Purdue  Scope:  cms  Name:  /store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/130005/48AF85C5-3DBD-BA46-9117-DA81D986F0BB.root
Traceback (most recent call last):
  File "deckard.py", line 451, in <module>
    pfns = lfns2pfns(rse_settings=attributes, lfns=lfns, operation='delete')
  File "/usr/local/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 226, in lfns2pfns
    return create_protocol(rse_settings, operation, scheme, domain, auth_token=auth_token, logger=logger).lfns2pfns(lfns)
  File "/usr/local/lib/python3.6/site-packages/rucio/rse/protocols/gfal.py", line 102, in lfns2pfns
    path = lfn['path'] if 'path' in lfn and lfn['path'] else self._get_path(scope=scope, name=name)
  File "/usr/local/lib/python3.6/site-packages/rucio/rse/protocols/protocol.py", line 360, in _get_path
    return self.translator.path(scope, name)
  File "/usr/local/lib/python3.6/site-packages/rucio/rse/protocols/protocol.py", line 258, in path
    algorithm_callable = RSEDeterministicTranslation._LFN2PFN_ALGORITHMS[algorithm]
KeyError: 'cmstfc'
[root@cms-consistency-int-rucio-consistency-tmp-deckard-54cdfd49xhfhh deckard]# echo $RUCIO_HOME
/opt/rucio/

a second run gives

[root@cms-consistency-int-rucio-consistency-tmp-deckard-54cdfd49xhfhh deckard]# python3 deckard.py -c /config/config.yaml  -r T2_US_Purdue
force_proceed is: False
Starting deckard for RSE:  T2_US_Purdue  with config:  <config.Config object at 0x7f9a5b0c3eb8>

 Scanner Output Path:  /var/cache/consistency-dump 
 minagedark:  4 
 maxdarkfraction:  0.02 
 maxmissfraction:  0.02 

Found scans for RSE:  T2_US_Purdue
36  unprocessed runs found for this RSE
Was the latest run /var/cache/consistency-dump/T2_US_Purdue_2021_06_28_20_47_stats.json attempted to be processed already?  True
Nothing to do here

So something is wrong with finding the TFC algorithm and then something is off with it declaring success before succeeding.

piperov commented 3 years ago

The second part is due to the fact that the run was already processed once, so it has the 'cc_dark', 'cc_miss' and 'cmp2dark' sections in the _stats.json file. Just delete those sections, and you will be able to re-run.

The first issue will need to be investigated.

ericvaandering commented 3 years ago

I solved this, but I think the logic is wrong. The stats should not be updated to say it processed correctly until it is. An error may be more useful