das queries with filters don't behave as expected

[neggert]

Hi,

I'm trying to run a DAS query with a filter and I'm getting an error. This works just fine on the web interface:

cms-sh|35> find file dataset=/DYToEE_M-10To20_TuneZ2_7TeV-pythia6/neggert-MCTSusy_Skim_Mar2012_DYToEE_M10To20-daf9758207e31d09324b57b0328014ab/USER | grep file.nevents>= 1

ValueError Traceback (most recent call last) /Users/nic/cms/MCTSusy/notebooks/Data/tW/ in () ----> 1 get_ipython().magic(u'find file dataset=/DYToEE_M-10To20_TuneZ2_7TeV-pythia6/neggert-MCTSusy_Skim_Mar2012_DYToEE_M10To20-daf9758207e31d09324b57b0328014ab/USER | grep file.nevents>= 1')

/Users/nic/cms/soft/install/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s, next_input) 1996 self._magic_locals = sys._getframe(1).f_locals 1997 with self.builtin_trap: -> 1998 result = fn(magic_args) 1999 # Ensure we're not keeping object references around:

2000 self._magic_locals = {}

/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.pyc in cms_find(arg) 137 Perform lookup of given query in CMS data-services. 138 """ --> 139 lookup(arg) 140 141 def cms_du(arg):

/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.pyc in lookup(arg) 159 res = apply_filter(flt.strip(), gen) 160 RESMGR.assign(res) --> 161 list_results(res, debug) 162 163 def verbose(arg):

/Users/nic/cms/soft/cmssh/src/cmssh/utils.pyc in list_results(res, debug) 77 return 78 if isinstance(res, list) or isinstance(res, GeneratorType): ---> 79 for row in res: 80 if not debug: 81 print row

/Users/nic/cms/soft/cmssh/src/cmssh/cmsfs.pyc in apply_filter(flt, gen) 66 def apply_filter(flt, gen): 67 """Apply given filter to a given set of results""" ---> 68 flt_func, flt_name = flt.split() 69 if isinstance(gen, list) or isinstance(gen, GeneratorType): 70 if flt_func == 'grep':

ValueError: too many values to unpack

[vkuznet]

Nic, cmssh does not have yet full DAS language. But you can do almost the same functionality via usage of results() function. Type your query find site dataset=/a/b/c then use python to loop over your results. Just see cmshelp for example. The results() will contains object which you can manipulate.

[neggert]

It looks to me like the only data that is returned for the query I'm doing is a list of files. Each file only has two attributes, assign and data. data is just a dict containing only the logical file name. Is there away to get at, say nevents?

Nic

On Tue, Apr 10, 2012 at 3:49 PM, Valentin Kuznetsov reply@reply.github.com wrote:

Nic, cmssh does not have yet full DAS language. But you can do almost the same functionality via usage of results() function. Type your query find site dataset=/a/b/c then use python to loop over your results. Just see cmshelp for example. The results() will contains object which you can manipulate.

---

Reply to this email directly or view it on GitHub: https://github.com/vkuznet/cmssh/issues/11#issuecomment-5054602

[vkuznet]

Yes, you can do it via info command. The find command just retrieve basic object, e.g. for dataset it is dataset name, for file it is file name. Then you need to invoke the info command over this. To do so for every file in a dataset you'll write

ip=get_ipython() for r in results(): ip.magic_info(r.data['logical_file_name'])

The query find file dataset=/a/b/c will return basic File object. Then you iterate over results and call info command. In other words it is replacement for manual steps:

find file dataset=/a/b/c info file1.root info file2.root

[vkuznet]

Ahh, sorry, just run this and output does not contain number of events, I need to add it.

[neggert]

It would be nice if the info command returned a dict so that I could do further operations on the information. Right now it doesn't look like it returns anything.

On Wed, Apr 11, 2012 at 9:54 AM, Valentin Kuznetsov reply@reply.github.com wrote:

Ahh, sorry, just run this and output does not contain number of events, I need to add it.

---

Reply to this email directly or view it on GitHub: https://github.com/vkuznet/cmssh/issues/11#issuecomment-5069156

[vkuznet]

It does. All commands are hooked back into results(). So if you invoke one print command, e.g.

info /store/generator/Summer09/Zee_M20_CTEQ66-powheg/GEN/MC_31X_V3_7TeV-v1/0000/E0375567-2331-DF11-AC84-000423D94D98.root

you'll get printout and your results will be saved, such that you can access them again via results(). So if you want to iterate, you need to modify the loop and invoke results() again.

For example

find file dataset=/a/b/c res = [r.data['logical_file_name'] for r in results()] res1=[] for r in res: ip.magic_info(r)
res1.append(results().data) for r in res1: print r.data, type(r.data)

[vkuznet]

Nic, if you need number of events for file, here is patch

--- a/src/cmssh/dbs2.py +++ b/src/cmssh/dbs2.py @@ -185,12 +185,13 @@ def block_info(block, verbose=None): return Block(blk)

def file_info(lfn, verbose=None):

• query = 'find file.name, file.size, file.createdate, file.createby, file.moddate, file.modby where
• query = 'find file.name, file.numevents, file.size, file.createdate, file.createby, file.moddate, params = {"api":"executeQuery", "apiversion": "DBS_2_0_9", "query":query} data = urllib2.urlopen(dbs_url(), urllib.urlencode(params)) rec = [f for f in qlxml_parser(data, 'file')][0] rec['logical_file_name'] = rec['file']['file.name'] rec['size'] = rec['file']['file.size']
• rec['nevents'] = rec['file']['file.numevents'] rec['created'] = time.strftime("%Y-%m-%d %H:%M:%S GMT", time.gmtime(rec['file']['file.createdate']) rec['createdby'] = rec['file']['file.createby'] rec['modified'] = time.strftime("%Y-%m-%d %H:%M:%S GMT", time.gmtime(rec['file']['file.moddate']))