eventually the index will outgrow a single JSON file, no?

BryanWilhite / songhay-ng-workspace

Angular workspace of shared libraries and ‘studio floor’ client

MIT License

1 stars 0 forks source link

eventually the index will outgrow a single JSON file, no? #13

Open BryanWilhite opened 6 years ago

BryanWilhite commented 6 years ago

what is the plan when the index grows to, say, 1GB or more?

what learnings are there out there to help you as you tread in deeper architectural waters?

BryanWilhite commented 6 years ago

paging the index by date and paging by category (and then by date) is the simplistic solution; this would require the generation of two sets of “sharded” (?) index snapshots

my ignorance insists that someone else smarter than me and this problem and did not resort to something like Redis cache or MongoDB

BryanWilhite commented 5 years ago

here is a new idea:

site/
    html/
    index/
    md/
    templates/

drilling into index/:

index/
    page-01.json
    page-02.json
    page-03.json
    ...
    page-<nn>.json

    groups/
        by-year-month/
            ...
            <yyyy>-<mm>.json

        by-topic/
            ...
            topic-*.json

every page/group JSON file will contain metadata about the entire set of pages or the entire group such that loading the first file allows loading of the other files on demand
the JSON files will hold arrays of MenuDisplayItemModel [docs]
index/page-*.json should come directly out of md/
index/ groups should come from page-*.json
the flexibility promise of 11ty should handle md/ to populate html/

BryanWilhite commented 5 years ago

New direction: I think the Index App should be based on lunr and Bing Web Search (or some open source equivalent).

The lunr folks on pre-building large indices:

This serialized index can then be written to a file, compressed, and served along side other static assets.

https://lunrjs.com/guides/index_prebuilding.html