documenting system

[gacafe]

• [x] high level overview of the system

ingest pipeline:

• [x] diagram of how ingest pipeline works
• [x] mention versioning of compact data

world:

• [x] diagram of how world / features / tags stuff works

search:

• [x] diagram how search works
• [ ] talk about graph weights stuff

for ingest pipeline / for each data source (binary)

• [x] what is the data
• [x] what does it do what other parameters it takes in
• [x] what should be the output
• [x] derivative things you need to run beyond the index
• [x] where do you grab the data?

[gacafe]

@xprogramari two separate things - one is a high level architecture diagram + compact and ingest pipeline interaction

[xprogramari]

i will start with general ingest pipeline / diagram !

[xprogramari]

(will send a separate cl for diagrams / readme stuff)

[xprogramari]

for the ingest binaries, i have double checked that each binary documents the following in flags -

• what is the data
• what does it do what other parameters it takes in
• what should be the output
• derivative things you need to run beyond the index

[xprogramari]

binaries also document type of data they need (e.g. GTFS / OSM etc) but they don't lead to links where these live online for example, or where they live in our cloud - i suggest we document that part in k8s pipelines, as that seems to make most sense / is how data ingestion will be run; i will add that to pipeline ticket