lun-4 / storcord

a badly-designed document store on top of discord (meme, not production ready, NOT PRODUCTION READY)
GNU Affero General Public License v3.0
23 stars 0 forks source link

document sharding #4

Open lun-4 opened 2 years ago

lun-4 commented 2 years ago

at the moment each document can only have 2048 characters, we should be able to expand over that limit.

we can do this by converting the document into a shard index once it goes over 2k chars, that index message contains all the message ids of the other shards, reassemble and kaboom you get a document

the index message itself might get over 2k chars too, so the database should handle multi level marketing multi level indexing so that we can infinitely scale to a gigabyte of data!!!!!!

lun-4 commented 2 years ago

basic support implemented in https://github.com/lun-4/storcord/commit/db50302bfbb4f1d9d2f18ade518fa6f21ac3d38a

multi level indexing not done