Relatively major feature request: keep list of valid categories for error flagging and to avoid duplication

[hschellman]

I was running tests and found that asking with runs[any] instead of core.runs[any] gives a python dump instead of an error message.

A very useful feature would be to maintain an internal DB of categories as they are added and then, if one gets this kind of error catch it and suggest that the person check that their category is in the list.

Not being able to generate a list of already defined parameters in sam has always been a pain. I think we could catch them as they go in and then be able to either ask the user to check the list or do it ourselves when we get a syntax error.

End users would love to have a list of the categories, even better would be if they were required to describe them.

Here is my example - my screwup but the error message was extremely uninformative.

metacat query -i "files from dune:all where core.file_type=detector and core.run_type='protodune-sp' and core.data_stream=physics and runs[any] in 5141:5200" > physics_ids.txt Traceback (most recent call last): File "/nashome/s/schellma/.local/bin/metacat", line 33, in sys.exit(load_entry_point('metacat==3.16.1', 'console_scripts', 'metacat')()) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_9_7/NULL/lib/python3/site-packages/metacat/ui/metacat_ui.py", line 91, in main cli.run(sys.argv, argv0="metacat") File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_9_7/NULL/lib/python3/site-packages/metacat/ui/cli/cli.py", line 205, in run self._run(command, context, argv, usage_on_error) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_9_7/NULL/lib/python3/site-packages/metacat/ui/cli/cli.py", line 200, in _run return interp._run(pre_command + word, context, rest, usage_on_error = usage_on_error) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_9_7/NULL/lib/python3/site-packages/metacat/ui/cli/cli.py", line 99, in _run return self(command, context, opts, args) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v3_9_7/NULL/lib/python3/site-packages/metacat/ui/metacat_query.py", line 118, in __call__ print("%s%s" % (f["fid"],meta_out)) TypeError: string indices must be integers metacat query -i "files from du

[ivmfnal]

If I run a query like that I get this error instead of the dump:

$ ui/metacat query files from dc4:dc4 where "runs[any] = 32"
Server side error: MQLExecutionError: File attribute runs value must be a scalar. Got array_any instead

Another related error message shows what attributes are allowed:

$ metacat query files from dc4:dc4 where runs = 32
Server side error: MQLExecutionError: Unrecognized file attribute "runs"
  Allowed file attibutes: creator, created_timestamp, name, namespace, size

I see that you run very old version of the client. Current versionis 3.17.3:

$ pip show metacat
Name: metacat
Version: 3.17.3
Summary: MetaCat is a general purpose metadata database. This package is the client side portion of the product.
Home-page: https://github.com/ivmfnal/metacat
Author: Igor Mandrichenko
Author-email: ivm@fnal.gov
License: BSD 3-clause
Location: /Users/ivm/miniconda3/lib/python3.7/site-packages
Requires: pythreader, pyjwt, requests
Required-by: datadispatcher

As for the list of categories, they are kept and viewable via GUI: https://metacat.fnal.gov:9443/dune_meta_demo/app/gui/categories/index

[ivmfnal]

Definition of category parameters is something an admin (or the collaboration) has to do. Current mechanism is via GUI.

[hschellman]

Yes but can we maintain a DB of them that one can query? Ie

metacat categories list

gives

core.runs array ownership description

or

metacat query “categories from “namespace” …."

or something like that.

Not having that in sam was a disaster. If people knew how to find categories they would not create so many and the UI could check for validity.

On Oct 12, 2022, at 9:40 AM, Igor Mandrichenko @.**@.>> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

Definition of category parameters is something an admin (or the collaboration) has to do. Current mechanism is via GUI.

— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fivmfnal%2Fmetacat%2Fissues%2F16%23issuecomment-1276457814&data=05%7C01%7Cheidi.schellman%40oregonstate.edu%7Cfe8bb131b0ba403bd44e08daac708966%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638011896580319099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OiVNKBppuQRxPwnGxMMHN5OJFEjJk1R1vfvGWTPOKl4%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIA37DOMAU5AEFAZLS5SSELWC3SZRANCNFSM6AAAAAARCOZY3I&data=05%7C01%7Cheidi.schellman%40oregonstate.edu%7Cfe8bb131b0ba403bd44e08daac708966%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638011896580319099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GgWMjjns0Nul%2FuNIzk9PUZJa%2Bh1xHZxqctbkBdX%2B%2FJw%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

[hschellman]

Like the gui!! But sometimes one needs to grep….

On Oct 12, 2022, at 9:38 AM, Igor Mandrichenko @.**@.>> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

If I run a query like that I get this error instead of the dump:

$ ui/metacat query files from dc4:dc4 where "runs[any] = 32" Server side error: MQLExecutionError: File attribute runs value must be a scalar. Got array_any instead

I see that you run very old version of the client. Current versionis 1.17.3:

$ pip show metacat Name: metacat Version: 3.17.3 Summary: MetaCat is a general purpose metadata database. This package is the client side portion of the product. Home-page: https://github.com/ivmfnal/metacat Author: Igor Mandrichenko Author-email: @.**@.> License: BSD 3-clause Location: /Users/ivm/miniconda3/lib/python3.7/site-packages Requires: pythreader, pyjwt, requests Required-by: datadispatcher

As for the list of categories, they are kept and viewable via GUI: https://metacat.fnal.gov:9443/dune_meta_demo/app/gui/categories/indexhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetacat.fnal.gov%3A9443%2Fdune_meta_demo%2Fapp%2Fgui%2Fcategories%2Findex&data=05%7C01%7Cheidi.schellman%40oregonstate.edu%7Cc76547b16a0942be0c7908daac7035d1%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638011895183203177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=T8wWoc6jf8FIA4qMyxxGuhGro9Q0LnKLAdcTWq9m0fk%3D&reserved=0

— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fivmfnal%2Fmetacat%2Fissues%2F16%23issuecomment-1276455227&data=05%7C01%7Cheidi.schellman%40oregonstate.edu%7Cc76547b16a0942be0c7908daac7035d1%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638011895183203177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iGpKXFmM13OHVcWavCsI0UG2duitaEABPLPAzR7%2F6T0%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIA37DPOU76ZKEJCJBD46TLWC3SQXANCNFSM6AAAAAARCOZY3I&data=05%7C01%7Cheidi.schellman%40oregonstate.edu%7Cc76547b16a0942be0c7908daac7035d1%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638011895183203177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TN2diNZ%2BXhkjoflQ%2BXH4KEX1%2FNeHzbVh%2FEjdd2amf6s%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

[ivmfnal]

Basic support for category listing and displaying is added to CLI and API. Please upgrade the client to 3.19.0. If additional functionality is needed, let's open a new issue.