helium / helium-packet-router

Apache License 2.0
8 stars 5 forks source link

config streaming #254

Closed michaeldjeffrey closed 10 months ago

michaeldjeffrey commented 10 months ago

SKF tables do not have write_concurrency because of the memory overhead. The process of "adding" a SKF involves removing other copies from the table with a select_delete/2. For sufficiently large tables of SKF this would bottleneck quite badly and slam all of the processing available.

Instead, on starting up the connection we don't try to remove any skfs, we know we'll receive only adds from the config service for the initial dump of the db, and there's a fairly consistent stream of removes coming in. Once we hit one of those it can be assumed we're done with the initial loading of data and we can go back to removing SKFs as they come in to be updated.