make it possible to read dataset from DBS3

ericvaandering commented 10 years ago

Original Savannah ticket 100976 reported by belforte on Sat Mar 23 06:36:57 2013.

starting point is this message from Marco Mascheroni:

-------- Original Message -------- Subject: Re: Coffee meeting today? Date: Thu, 21 Mar 2013 10:03:11 +0100 From: Marco Mascheroni <marco.mascheroni@cern.ch> To: Daniele Spiga <daniele.spiga@cern.ch> CC: Eric W Vaandering <ewv@fnal.gov>, Stefano Belforte <stefano.belforte@cern.ch>, Maria Girone <Maria.Girone@cern.ch>, "Federica Fanzago" <federica.fanzago@cern.ch>, Stefano Belforte <stefano.belforte@ts.infn.it>, Hassen Riahi <hassen.riahi@pg.infn.it>, "Diego da Silva Gomes" <diego@cern.ch>

Hi,

the work I did for CRAB3 is in this patch:

https://github.com/mmascher/WMCore/commit/56a410bdec25bee5ebd53016c13beb30aa9bad8d

You can find there the implementation of a WMCore API for both DBS2 and DBS3 readers. I called it listDatasetFileDetails. There is also a unit test for it.

Given a dataset this API returns a list whose elements are dicts (one per file). For each file you have something like:

         { 'NumberOfEvents': 545,
           'BlockName': '/HighPileUp/Run2011A-v1/RAW#dd6e0796-cbcc-11e0-80a9-003048caaace',
           'Lumis': {173658: [8, 12, 9, 14, 19, 109, 105] ...},
           'Parents': [],
           'Checksums': {'Checksum': '22218315', 'Adler32': 'a41a1446', 'Md5': 'NOTSET'},
           'Size': 286021145
         }

ericvaandering commented 10 years ago

Comment by belforte on Wed Apr 24 09:22:55 2013

update: still no news on DBS2-DBS3 migration plan. Production is currently writing new dataset in both, but we (CMS Computing) have not made any decision on when stopping to write in DBS2, when to start putting user's publication in DBS3, when to retire DBS2. No news of any work so far on code for crab publication in DBS3. Stefano thinks that most likely scenario is still that DBS2 is retired in september 2013.

ericvaandering commented 10 years ago

Comment by belforte on Mon Sep 23 10:30:10 2013

latest rumor I heard from DMWM is that DBS3 rolling in is "january". There was something also about DBS2 stopping to work in Novermber when SHA-2 certs come into the game, but Eric did not know anything about this. In the meanwhile I am converging on using DAS here and tracking understanding/progress in a twiki page https://twiki.cern.ch/twiki/bin/view/CMS/FromDBS3toCrab2

belforte commented 10 years ago

fix released in 2_10_1

dmwm / CRAB2

make it possible to read dataset from DBS3 #893