Index configuration

[elv-zenia]

Create a new page for managing index fields with an additional way to update search index.

Index settings

• site object (root) - a picker for an object

fabric:
  root:
    content:
        "iq__3eJ4QHB7R8wmmvhp4Hz7eqRZfzg"
     library:
        "ilib2XX6yS9S8bgAeLVxDGKeoNcNVckN"

• document prefix (text) default /

indexer:
    arguments:
        document:
            prefix:
                "/assets/*"

• query suffix (a term that is always added to the end of all queries)
  • maybe 'static query terms'

indexer:
    arguments:
        query:
             suffix:
                   "AND (-f_original_access:[100 TO 300])"

Example structure of indexer fields:

"indexer": {
      "config": {
        "indexer": {
          "arguments": {
            "fields": {
              "logo": {
                "options": {
                  "stats": {
                    "histogram": true
                  }
                },
                "paths": [
                  "site_map.searchables.*.video_tags.metadata_tags.*.metadata_tags.shot_tags.tags.text.Logo.text"
                ],
                "type": "text"
              },
              "title_type": {
                "options": {},
                "paths": [
                  "site_map.searchables.*.video_tags.metadata_tags.*.metadata_tags.shot_tags.tags.text.title"
                ],
                "type": "string"},
            }
          }
        }
      }
    }

Steps for updating index:

1. Edit content and save write_token
2. Call bitcode search_update
3. Finalize content

Search reference: https://github.com/qluvio/elv-search

[elv-zenia]

Pre-configuration:

• Within index content type, set public/title_configuration/show_indexer_settings value to true. There is also a corresponding setting in the UI via the spec form (gear icon) for the content type.

To verify: NOTE: make sure that the config url being used is a search node.

1. Add a new field via the + icon.
2. Remove a field via the trash icon.
3. Save with newly added/removed fields and ensure metadata is updated.
4. Saving without entering a commit message should have a commit of "Update index configuration".

Ready on https://core.test.contentfabric.io/prod/ core.demov3

[elv-serban]

STEP 2

If the indexer has run already, display some information:

• info on when it was last run (if we don't have it we can simply say if it was never run)
• stats
  • number of documents
  • number of fields indexed
    • if not too cluttered, a table of the fields and total documents per field
• errors - there is an 'exceptions' key in the resulting JSON

Also offer a search box called "Terms" (note in a future version we would like to have more search parameters but starting with just terms)

We need to decide how best to display the results - the 'tree' control we use in "Show metadata" is probably the best first.

{
  "indexer" : {
     "stats" : {
        "document": {
             "total": 100
        },
        "fields" : {
            "f_asset_type" : {
                  "value" : {
                      "total" : 10,
                       // the histogram fields have more info
                   }
             }
        }
     }
  }
}

[elv-zenia]

Issue reported by SS on slack:

• Stop crawl_status loop when run_state is failed

[elv-zenia]

Issues reported by SS on slack 3/27:

• Add "type": "metadata-text" unconditionally at path /indexer/config/indexer/type
• Query field should be empty string when there is no user-provided value (path indexer/config/indexer/arguments/query)
• Path field should have the following placeholder text: site_map.searchables.*.title
• Histogram checkbox field should be underneath Type dropdown
• Crawl status should stop at state 'terminated' as well as run_state 'failed'
• Weird behavior when deleting config fields. Deleting the second time and beyond doesn't remove the field

Above fixed by https://github.com/eluv-io/elv-fabric-browser/pull/81/files/fb9e2709eb98f5907718f02abd1b2c4c58bf0eb0..df725a29e97dcf418bed10288f6a5d1d1b5fe14d

[elv-serban]

Suggested in-app help

Help text

Site Object

   Select the "site" object containing the list(s) of titles or other media and searchable objects.

Document Prefix

  The object metadata path at which the searchable document is defined.  Typically "/" to use the entire object.  
  Other common examples:
     /assets/*      (for individual assets within a title object)

Query Suffix

  A term that is automatically appended to all queries.

Configuration Fields

 The list of metadata fields to be indexed and available as search terms.  Each field has a label (for example 'title', 'synopsis', 'release_date', ...) and one or multiple metadata paths pointing to the metadata fields that contain the values for this field.

Label:

 The label of his searchable field (for example 'title', 'synopsis', 'release_date', ...).  This table can be used in search queries prefixed with 'f_' (for example 'f_title', 'f_synopsis', 'f_release_date', ...).

Paths:

One or multiple metadata paths pointing to the metadata fields that contain the values for this field.
Common examples:

        public.asset_metadata.titles.*.*.title
        public.asset_metadata.series.*.*.episodes.*.*.title

        public.asset_metadata.titles.*.*.info.synopsis
        public.asset_metadata.series.*.*.episodes.*.*.info.synopsis

When using the site 'searchables' feature:
       site_map.searchables.*.asset_metadata.title
       site_map.searchables.*.assets.*.title

Type:

    string  - the field is matched in entirety (suitable for tags and titles)
    text     - the field is matched by individual words (suitable for synopsis and descriptions)

Histogram:

   Return a histogram of all matched values.  Only available for type 'string'.

Pre-fill text for text boxes

Label: title (e.g.)

Paths: public.asset_metadata.titles...title (e.g.)

[elv-zenia]

Deployed changes for

• Form label help text
• Set query to empty object when no value is provided

[elv-zenia]

add a 'Search Index" box to the app. If there is an index (if last run is successful) then we can add a third section and just call it "Search Index" with a text box. We will add some sample search terms in the '?' inline help. Then click "Search" and just make the search request and display the result in that json tree format (same as you do currently with the search stats).

[elv-zenia]

Deployed to prod.