[Epic] Full text search

[zelima]

As a Consumer looking for data I want to get a dataset on topic X so that I can use it for my work

As a Consumer looking for data I want to be able to search with relevant terms and see if there are datasets available that are related

• I want best matches in terms of topic
• I prefer better quality data (e.g. core datasets) to less good (or less reputable people)

What's the situation now:

• Have search page
• But just searching dataset titles - and should search README too (but higher weighting for title)
  • keywords??
• core datasets come last after alternatives ...

Acceptance Criteria

• [ ] Search for finance brings up vix? Search for climate brings up co2-ppm
  • [x] Searching for terms in the README works
• [x] Search for a username should bring up their datasets (?:)
• [x] core datasets get preference ... (so top of the list by default e.g. with empty query)

Tasks

• [x] Do analysis
  • [x] How does the system work (start from frontend and work backwards)
  • [x] Run elasticsearch on local server [0.5d]
  • [x] analyze metastore api
  • [x] figure out how tests work
  • [x] figure how stuff is added to the ES index
• [x] Modifying the frontend (?): nothing to do
• [x] Modifying search API (in metastore repo):
  • [ ] (minor) add docs to query to document parameters - mini separate PR)
  • [x] (minor) code may need refactoring, build_dsl is ugly (how?) and passing "kind" around seems bad smell (kind is a switch on the table to search i.e. datasets or events)
    • WONTFIX - works fine for now
  • [x] write tests where readme would be relevant
  • [x] BUT TESTS USE OWN MAPPING DIFFERENT FROM REAL MAPPING!!
  • [x] alter the query to ES backend to use multi-match query (?) https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html
  • [x] Change to use the current real mapping in the tests
  • [x] ? move controller tests to model tests and have a trivial controller tests (just make sure that works) ...
  • WONTFIX by testing controller we are actually testing models (as controller is simple caller function here: https://github.com/datahq/metastore/blob/master/metastore/controllers.py#L6)
  • [ ] core datasets get preference https://github.com/datahq/metastore/issues/25
• [x] Adding datasets to ES index (and structure of the index):
  • [x] move from assembler.dump_to_s3 to specstore (do mapping using tableschema_elasticsearch) https://github.com/datahq/specstore/issues/28
  • [x] change mapping and allow to search in datapckage.readme
  • [x] re-index via kibana and change aliases

Analysis

Search system has several parts:

• [Querying and Displaying in frontend] - nothing to change here (?) - https://github.com/datahq/frontend/blob/master/lib/index.js#L128
• API for Querying ES index - linked the docs for this web API https://github.com/datahq/metastore#api and the code for this API https://github.com/datahq/metastore/blob/master/metastore/models.py#L104
  • across different fields at the same time
• Adding datasets to ES index (with relevant properties available for searching) => see detailed separate analysis section below

Problem: we don't index the readme atm so we can't search it

=> to change indexing needs changing mapping (or moving the readme into datahub.description) => either of these need a reload of the datapackages => editing dump to s3 assembler + a rerun of all flows [painful and complex] => a deeper analysis of the issue => should we re-architect a bit

Solutions:

• Write readme into description field - somewhere in https://github.com/datahq/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py#L47-L61
  • No remapping needed
  • Need to reindex datasets (rerun pipelines) or wait until pipelines run again ??
  • (+) least invasive change ...
  • (-) continues ugliness of having the metastore indexing done in the pipelines (and only works because description is indexed)
• index readme (separate from description) but not re-archictecture
  • Remapping needed => in assembler
  • Need to change assembler and search API to search readme too
  • Need to reindex all datasets (rerun pipelines)
  • OR: we can do mapping change direct now and try and do an update_by_query - see https://www.elastic.co/blog/reindex-is-coming
  • No change in frontend
• rearchitect
  • dataset indexing in metastore (or own library)
  • ES index creation in metastore (how do we ensure happens - can just check every time as we currently do)
  • either call that from pipelines or from spectore or implement a subscriber model
  • TODO: understand how events work ...
  • implement reindex system and script so we can rerun as needed in the future

Questions

• Are there other changes to the search we will need to do in the future? Maybe we should also search
  • datahub.licenses
  • title (text - with boosting)
  • description and readme (text)
  • bytes
  • resources.title and resources.description
• Can you not index an object generally but index subfields and how using ES?
  • ??
• How do you do "database migrations" with ES i.e. change a mapping?
  • Cannot change mapping of existing fields but maybe can add fields
  • To reindex is possible (maybe) ... - https://www.elastic.co/blog/reindex-is-coming
  • Use aliases to have versions of an index
  • reindex from one index to another
  • change the alias to point to new index

How the load to metastore works today

2 parts

• Set up of the mapping and index
• Adding documents

Where and when does ES index get set up?

• Where: https://github.com/datahq/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py#L17
• When: it gets set the first time the first dataset gets run through the flow https://github.com/frictionlessdata/datapackage-pipelines-elasticsearch/blob/master/datapackage_pipelines_elasticsearch/processors/dump/to_index.py#L70
  • Which calls https://github.com/frictionlessdata/tableschema-elasticsearch-py/blob/master/tableschema_elasticsearch/storage.py#L93
• Adding documents (datasets) to it

Adding documents

graph LR

dp[Data Package] --> dumptos3(Dump to S3 Pipeline)
dumptos3 --> weirddp[DP with single resource<br/>with a single row that is the object for ES]
weirddp --> dumptoes(Write to ES Pipeline)

• Start with a normal Data Package that has completed dump to s3

• Convert this data package into a new "data package" DP2 - https://github.com/datahq/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py

  • DP2 is not a "real" data package but just a way to pass arguments to the next processor

  • DP2: has a single resource with one row that is

    'id', 'name', 'title', 'description', # readme was not pulled out into it dp.readme or dp.description 'datahub' 'datapackage': # descriptor minus resources

  and a schema that is the ES schema

• The ES dumper which is built around tableschema dumper then dumps this "data package" to ES
  • This is kind of weird as ES dpp is about pushing a tabular resource to ES

what's not working about this

• We (ab)using ES dumper - it is designed for "real" resources with a table structure
• We attempt to write the mapping every time (minor)
• The mapping and other ES set up is not in the metastore nor even in ES pipeline but in dump to s3!
  • => fragility if we changed the planner structure
  • => very confusing for developers (and to alter metastore i have to get into dpp)
• Not clear what we do if we want to reindex in some way - we have to rerun all pipelines?
  • We can't just reindex we have to reprocess all data.

How to re-architect

graph TD

raw[Raw Data] --> factory[Factory]
factory --> finished[Finished Product: Data Package on BitStore]
finished -.done.-> foreman[Foreman]
foreman --> indexer[Indexing Service]
indexer --put--> metastore[MetaStore]

Do we push notify the ES index system or does the index system listen?

Requirements:

• We can change the metastore dataset index. And can reindex the metastore without re-running the entire data factory
  • Is it a big deal re-running after all nothing has changed as no new data or no new ...
  • How does Factory decide what to rerun? e.g. if datapackage.json has just changed does it rerun csv generation or not?
• We
• [Generally: We can add new exports e.g. to bigquery without re-running the entire data factory - that sort of happens already i think because source will not have changed?]

Going forward

• metastore dataset "index" library that:
  • creates the index (with mapping)
  • adds a document to that index
• Events API (python): https://github.com/datahq/events/blob/master/events/events.py

https://github.com/datahq/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py#L17

Repos to work with:

• Search logic: https://github.com/datahq/metastore/blob/master/metastore/models.py
• Indexing the ElasticSearch: https://github.com/datahq/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py

[Mikanebu]

Could we consider search in dashboard page? There is a situation when you cannot find dataset on users dashboard's search box, but it can be found on the main search page. Is this related to this issue or i need to create separate issue?

[zelima]

@Mikanebu no this is not related. Please open separated issue in frontend repo

[zelima]

FIXED.

• We moved from dump_to_s3 to specstore for now
• datapackage added to mapping
• analyzers added for fields
• boosting title and README
• reindexed without downtime

Finance and climate present neither in title nor in README so none of them result as expected. That may be probably solved with keywords. But not related to this issue

• core datasets get preference move here: https://github.com/datahq/metastore/issues/25