exact string match required for tags

[ndushay]

4 out of 5 of the reported desired search functionality issues for the argo 3 rollout (see #669) were about tag searching.

From Ben: "DPG staff have started to populate an example list with examples of old searches and desired results: https://docs.google.com/document/d/1h0wcEZ0SHsFnIcQ3tDmtojezJ8bw7DAMYaHASKLuuL0/edit "

case 1: DIGREQ-820 (also DIGREQ-916)

• The query string is a jira ticket number.
• this information is stored in a tag in identityMetadata as "JIRA : DIGREQ-820" (but no quotes)

<identityMetadata>
  <sourceId source="sul">M1999_fb_55</sourceId>
  <objectId>druid:fg572cf3879</objectId>
...
  <tag>Project : Digitization Request</tag>
  <tag>JIRA : DIGREQ-820</tag>

• workaround: "JIRA : DIGREQ-320" as quicksearch string: quotes required, exact string (caps, JIRA space colon space prefix)
• workaround 2: find one object with this tag link and click on it: https://argo.stanford.edu/catalog?f[tag_ssim][]=JIRA+%3A+DIGREQ-820

  case 2: Beasts and Books

• The query string is an exhibits name
• this information is stored in a tag in identityMetadata as "DPG : Exhibits : Beasts and Books" (but no quotes)

<identityMetadata>
  <sourceId source="sul">BARCH_QH41_A4_F_V6</sourceId>
  <objectId>druid:bp318nw6601</objectId>
...
  <tag>Project : Special Collections - Exhibitions</tag>
  <tag>DPG : Exhibits : Beasts and Books</tag>

• workaround: "DPG : Exhibits : Beasts and Books" as quicksearch string: quotes required, exact string (exact case, DPG space colon space Exhibits space colon space as prefix)
• workaround 2: find one object with this tag link and click on it: https://argo.stanford.edu/catalog?f[tag_ssim][]=DPG+%3A+Exhibits+%3A+Beasts+and+Books

  case 3: Stanford Publications

• the query string is a project name
• this information is stored in a tag in identityMetadata as "Project : Stanford Publications" (but no quotes)

<identityMetadata>
  <sourceId source="sul">8050_56_1954</sourceId>
  <objectId>druid:bh303xg8617</objectId>
...
  <tag>Project : Stanford Publications</tag>
  <tag>DPG : Stanford Publications : Stanford Review</tag>

• workaround: "Project : Stanford Publications" as quicksearch string: quotes required, exact string (exact case, Project space colon space as prefix)
• workaround 2: find one object with this project link and click on it: https://argo.stanford.edu/catalog?f[project_tag_ssim][]=Stanford+Publications

  case 4: ars0021

• query string a project number
• this information is a collection number for the Stanford Recital Collection; it is in a MODS note field in the descMetadata in the APO object. (cj857zs2426).

<mods>
  <note>Related collections at the Archive of Recorded Sound include
 the Stanford Recital Collection (ARS0021), Stanford Speech Collection (ARS0022), 
...
Stanford Marching Band Collection (ARS0089), 
the Transcription/Instantaneous Disc Collections and the Harold Schmidt Collection (unprocessed).</note>

• this information is also a project tag and in the sourceid for member items in the collection.

<identityMetadata>
  <sourceId source="sul">ars0021_b119_19991206</sourceId>
  <objectId>druid:bb980rx0621</objectId>
...
  <tag>Project : ARS0021 recital : dat</tag>

• workaround: "Project : ARS0021 recital : dat" as quicksearch string: quotes required, exact string (exact case, Project space colon space ARS0021 recital space colon space dat)
• workaround 2: find one object with this tag link and click on it: https://argo.stanford.edu/catalog?f[tag_ssim][]=Project+%3A+ARS0021+recital+%3A+dat
• workaround 3: find one object with this project link and click on it: https://argo.stanford.edu/catalog?f[project_tag_ssim][]=ARS0021+recital+%3A+dat

[ndushay]

In argo3, tags are

• indexed, as is, to tag_ssim, which is a String field.
  • see https://github.com/sul-dlss/dor-services/blob/master/lib/dor/datastreams/identity_metadata_ds.rb#L16
  • tag_ssim is included in the qf param sent with the query, but:
  • Exact match, case and whitespace sensitive, with quotes, is needed.
• indexed, minus their prefix, to (prefix)_tag_ssim
  • see https://github.com/sul-dlss/dor-services/blob/master/lib/dor/datastreams/identity_metadata_ds.rb#L107-L115
  • (prefix)_tag_ssim is not included in the qf param sent with the query, so (prefix)_tag_ssim field is not used by quicksearch.
  • Exact match, case and whitespace sensitive, with quotes, will be needed.
• indexed in chunks to exploded_tag_ssim, also a String field.
  • see https://github.com/sul-dlss/dor-services/blob/master/lib/dor/datastreams/identity_metadata_ds.rb#L117-L126
  • chunks are from beginning, so A : B : C is indexed as A, A : B, and A : B : C
  • since JIRA is a prefix, searching for this tag without "JIRA : " prefix will never be successful.
  • exploded_tag_ssim is not included in the qf param sent with the query, so exploded_tag_ssim field is not used by quicksearch.
  • Exact match, case and whitespace sensitive, with quotes, will be needed.

Besides the workaround of using the exact string to match as a quoted search, some ideas:

• also index tags as tokenized, and include them in searched quicksearch fields passed to Solr in qf param.. Requires reindexing and Solr config changes. (basically, only tokenize it if it won't be a facet, do both tokenized and string if it is a facet)
• return of the tokenized catchall field (requires reindexing) AND bump up mm values so dismax searches will behave like AND for more query terms (see #685). Requires reindexing and Solr config changes.

[ndushay]

Argo 3 has edismax query parser in the config, but passes in defType=dismax.

module BlacklightConfigHelper
  def self.add_common_default_solr_params_to_config!(config)
    config.default_solr_params = {
      :'q.alt' => '*:*',
      :defType => 'dismax',

and uses these fields for qf, also configured somewhere in argo:

  "qf"=>"text^3
    creator_tesim
    dc_creator_tesim
    dc_identifier_druid_tesim
    dc_title_tesim
    dor_id_tesim
    extent_ssim
    identifier_tesim
    lifecycle_ssim
    obj_label_tesim
    obj_state_tesim
    originInfo_place_placeTerm_tesim
    originInfo_publisher_tesim
    public_dc_contributor_tesim
    public_dc_coverage_tesim
    public_dc_creator_tesim
    public_dc_date_tesim
    public_dc_description_tesim
    public_dc_format_tesim
    public_dc_identifier_tesim
    public_dc_language_tesim
    public_dc_publisher_tesim
    public_dc_relation_tesim
    public_dc_rights_tesim
    public_dc_subject_tesim
    public_dc_title_tesim
    public_dc_type_tesim
    scale_ssim
    source_id_ssim
    tag_ssim
    title_tesim
    topic_tesim",

(note dc stuff: YUCK)

[ndushay]

closed in favor of #4265