search

kmgreen2 / agglo

Agglo: A Process-Anywhere Framework for Event Stream Processing
https://kmgreen2.github.io/agglo/
MIT License
2 stars 0 forks source link

Lineage-Based Search Engine #56

Open kmgreen2 opened 3 years ago

kmgreen2 commented 3 years ago

Can probably POC this using elasticsearch (https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html).

Basic idea:

  1. Have N pipelines consuming events (news sites?) from N sources that are all anchored to the same ticker
  2. Index the events in elastic
  3. Update the index with a number derived from the "happened before" relation to sort.
kmgreen2 commented 2 years ago

Added basic elastic support. Still todo:

  1. Tee process that puts a document into elastic
  2. Add example pipeline that includes ...Entwine -> Elastic Tee...
  3. Create utility that fetches docs from elastic and verifies the ordering