search

sourcenetwork / defradb

DefraDB is a Peer-to-Peer Edge Database. It's the core data storage system for the Source Network Ecosystem, built with IPLD, LibP2P, CRDTs, and Semantic open web properties.
398 stars 40 forks source link

[Epic] Secondary Indexes #201

Open jsimnz opened 2 years ago

jsimnz commented 2 years ago

Currently, only Primary Indexes are implemented, which is a simple relation between dockey/field => state_data. Effectively this is an index on the dockey field of a document. This is a minimal requirement for making the DB work, however we need to let devs also manually define "Secondary Indexes".

Secondary Indexes allow devs to index on alternative fields, even a combination of multiple fields. Which can lead to effecient querys for certain filter/order/etc arguments. The idea is if a developer knows ahead of time the likely kind of queries/lookups their respective application needs, then they can optimize the DB structure for those specific queries.

Starting a spec doc as there are a number of goals/compromises/designs that need to be considered: https://hackmd.io/rLHMu-KvRFC0OoE8xkwYlA

- [x] https://github.com/sourcenetwork/defradb/issues/231
- [ ] https://github.com/sourcenetwork/defradb/issues/297
- [ ] https://github.com/sourcenetwork/defradb/issues/1553
- [ ] https://github.com/sourcenetwork/defradb/issues/1555
- [ ] https://github.com/sourcenetwork/defradb/issues/298
- [ ] https://github.com/sourcenetwork/defradb/issues/299
- [ ] https://github.com/sourcenetwork/defradb/issues/296
- [x] #307
- [ ] https://github.com/sourcenetwork/defradb/issues/1585
- [ ] https://github.com/sourcenetwork/defradb/issues/2132
- [ ] https://github.com/sourcenetwork/defradb/issues/2229
- [ ] https://github.com/sourcenetwork/defradb/issues/2180
- [ ] https://github.com/sourcenetwork/defradb/issues/2247
- [ ] https://github.com/sourcenetwork/defradb/issues/2174
- [ ] https://github.com/sourcenetwork/defradb/issues/2279
- [ ] https://github.com/sourcenetwork/defradb/issues/2280
- [ ] https://github.com/sourcenetwork/defradb/issues/2601
- [ ] https://github.com/sourcenetwork/defradb/issues/2680
AndrewSisley commented 2 years ago

Had a thought - if/when implementing the ds.Get call chain, do it in a feature-agnostic way that allows fetcher to use ds.Get when it can