search

cernopendata / cernopendata-client

CERN Open Data command-line client
http://cernopendata-client.readthedocs.io/
GNU General Public License v3.0
10 stars 9 forks source link

Searching for a record from the title #87

Open avivace opened 3 years ago

avivace commented 3 years ago

I'm trying to understand how the search from the title is supposed to work. Is there a way to also specify (part of) the additional_title key? If not, the search seems to often just throw More than one record fit this title.This should not happen.

This is what I'm trying:

from cernopendata_client import searcher

SERVER_HTTP_URI = "http://opendata.cern.ch"

# Check if record with the given recid exists
searcher.verify_recid(server=SERVER_HTTP_URI, recid=1)

metadata_from_recid = searcher.get_record_as_json(server=SERVER_HTTP_URI, recid=1)

metadata_from_doi = searcher.get_record_as_json(server=SERVER_HTTP_URI, doi=metadata_from_recid["metadata"]["doi"])

metadata_from_title = searcher.get_record_as_json(server=SERVER_HTTP_URI, title=metadata_from_recid["metadata"]["title"])

print(metadata_from_recid == metadata_from_doi == metadata_from_title)
tiborsimko commented 3 years ago

For some "unique" records it works well:

$ cernopendata-client get-metadata --title '/SingleElectron/Run2011A-v1/RAW'

However for many records the title is also referenced in other records, e.g. in Configuration files and elsewhere, which is why more than one record is found.

What we should do is to specify exact matching in a wanted field. We have actually an open issue about better --title matching, see #56.

This may need to (1) expose facets and indexes to the client-side command line so that people could specify things like --facet Dataset/Derived, or we might need to (2) alter some Elasticsearch mappings and analyzers on the server side to create better indexes for the exact and fuzzy searches to work better.

For example, WRT option (1), until we have nicer indexes on the server-side, our best bet would be to get the REST API JSON response and instead of returning "More than one record fit..." error message, go through the JSON output and do additional parsing to achieve what was wanted by the user (e.g. return only derived datasets, not configuration file records).

Furthermore, (3) I was going to open a new issue about searching metadata in any field, not only in titles. This comes from a CMS Open Data use case. So perhaps we could consider all this at once and address better searching together?