documenting system - Githubissues

diagonalworks / diagonal-b6

b6: Diagonal's geospatial analysis engine.

Apache License 2.0

24 stars 3 forks source link

documenting system #281

Open gacafe opened 3 months ago

gacafe commented 3 months ago

[x] high level overview of the system

ingest pipeline:

[x] diagram of how ingest pipeline works
[x] mention versioning of compact data

world:

[x] diagram of how world / features / tags stuff works

search:

[x] diagram how search works
[ ] talk about graph weights stuff

for ingest pipeline / for each data source (binary)

[x] what is the data
[x] what does it do what other parameters it takes in
[x] what should be the output
[x] derivative things you need to run beyond the index
[x] where do you grab the data?

gacafe commented 3 months ago

@xprogramari two separate things - one is a high level architecture diagram + compact and ingest pipeline interaction

xprogramari commented 3 months ago

i will start with general ingest pipeline / diagram !

xprogramari commented 2 months ago

(will send a separate cl for diagrams / readme stuff)

xprogramari commented 2 months ago

for the ingest binaries, i have double checked that each binary documents the following in flags -

what is the data
what does it do what other parameters it takes in
what should be the output
derivative things you need to run beyond the index

xprogramari commented 2 months ago

binaries also document type of data they need (e.g. GTFS / OSM etc) but they don't lead to links where these live online for example, or where they live in our cloud - i suggest we document that part in k8s pipelines, as that seems to make most sense / is how data ingestion will be run; i will add that to pipeline ticket