Set up analyzers

[jmartinm]

• [ ] Take into consideration LaTeX formulas. Titles and abstract can contain LaTeX formulas that should not be broken by the tokenizer so that they are searchable. A custom analyzer could be used.
• [ ] Experiment. Have an analyzed a non-analyzed version to match CERN-LHC-LHCb even when the user has typed LHCb.
• [ ] We have to make sure, at record creation/modification, that the DOI is normalized e.g. to 10.xxx/xxx, without any superflous prefix, and that the search term is receiving the same treatment. Alternatively we need, for every DOI to index it also with doi: and http://dx.doi.org/ prefixes. This seems something that could be made easily with a Groovy script. Or altenatively with one more pythonic enhancer.
• [ ] arXiv eprints. Same as with DOIs they need to be found with and without prefix.
• [ ] isbns Keep raw and analyzed version to find them with and without dashes.
• [ ] Create another field for title and abstract where stemming is applied. The boost order (from more boosting to less) should be: 1) Raw 2) Basic Analyzer 3) Stemmed. A stemmer needs to be chosen - probably either light_english or minimal_english and some words can be protected from stemming.
• [ ] Expand title, abstract and keywords with synonims. (See: https://www.elastic.co/guide/en/elasticsearch/guide/current/using-synonyms.html)

[jmartinm]

Tasks extracted from https://github.com/inspirehep/inspire-next/pull/641

[kaplun]

This seems something that could be made easily with a Groovy script. Or altenatively with one more pythonic enhancer.

When possible, :+1: to indeed use index-time expansions + serch-time normalization based on Groovy!

[kaplun]

Expand title, abstract and keywords with synonyms.

@fschwenn the HEP.rdf ontology is still maintained at DESY, correct? Do you think it can be used as a basis for building a synonyms table? (i.e. simple string mapping useful for indexing/searching a given word as two words).

[fschwenn]

Yes, the ontology is still maintained at DESY. It contains informations on synonyms in the sense that if altLabel is found, prefLabel is assigned. But one can not use it without proof reading, e.g.

string: Green-Schwarz Green-Schwarz action Green-Schwarz superstring

If "Green-Schwarz action" is found in the text, we assign "string: Green-Schwarz". However, they are not really synonyms.

Florian Schwennsen Deutsches Elektronen-Synchrotron DESY Building 01 Room O1.446 phone: +49-40-8998-6190

From: "Samuele Kaplun" notifications@github.com To: "inspirehep/inspire-next" inspire-next@noreply.github.com Cc: "Florian Schwennsen" florian.schwennsen@desy.de Sent: Thursday, 7 January, 2016 11:07:37 Subject: Re: [inspire-next] Set up analyzers (#685)

Expand title, abstract and keywords with synonyms. @fschwenn the HEP.rdf ontology is still maintained at DESY, correct? Do you think it can be used as a basis for building a synonyms table? (i.e. simple string mapping useful for indexing/searching a given word as two words). — Reply to this email directly or view it on GitHub .

[kaplun]

Yes, the ontology is still maintained at DESY. It contains informations on synonyms in the sense that if altLabel is found, prefLabel is assigned. But one can not use it without proof reading.

I see... @inspirehep/inspire-dir do you have maybe a good idea were we could obtain a HEP synonym list? E.g. to be able to search for β and beta, for "HEP" and for "High Energy Physics", and of course much more sensible substitutions...

[jmartinm]

Not sure if this can be helpful :) http://www.personal.kent.edu/~plucasst/HEPThesaurus/Alphabetical%20Thesaurus.htm

[ksachs]

I think SPIRES had a list at least for the most common abreviations. E.g. title-variant supersymmetry got automatically assigned to articles with title SUSY.

[kaplun]

@jacquerie wanted to attack mappings at some point so I am assigning to him so that we are sure not to forget the points listed here.

[kaplun]

WRT LaTeX see also #1165

[jmartinm]

@StellaCh this should be reviewed by the person implementing the search since it seems still relevant.

[chris-asl]

@jmartinm regarding DOI and arXiv eprints, what could the superflous prefix be? Something like arxiv:ARXIV_ID? And for DOIs?

[jmartinm]

@jmartinm regarding DOI and arXiv eprints, what could the superflous prefix be? Something like arxiv:ARXIV_ID? And for DOIs?

Ideally you can check the search logs for examples. For arxiv the prefix that some people might use is arxiv: and for DOI it could be doi: or http://dx.doi.org/ @annetteholtkamp or @michamos can confirm if they know other prefixes that should work when searching

[michamos]

@chris-asl for arXiv we have https://github.com/inspirehep/inspire-schemas/blob/ac809becb858cfe2a1c695c18f00014109138dca/inspire_schemas/builders/references.py#L110-L126. You can move it to inspire_schemas.utils if it's needed for search too. For dois, you could use directly idutils.normalize_doi.