search

cefn / watchable

Repo for @watchable/store and supporting packages.
MIT License
5 stars 1 forks source link

Consider removing @watchable/queue from nevermore #84

Open cefn opened 1 month ago

cefn commented 1 month ago

Currently @watchable/queue is used by nevermore to sequence job settlements. The implementation of @watchable/queue employs immutable arrays (meaning every insertion copies items into a new array). This could lead to a significant performance impact when there is a backlog of e.g. 1000-plus items. It may explain why bulk tests slow down after e.g. 10000 items

A better design would be to employ a FIFO queue employing a linked list with head and tail pointers. This has an O(1) insertion and removal cost regardless of the length of the queue.

cefn commented 1 month ago

Another related intervention might be to switch @watchable/queue to use FIFO internally and find a different way to introduce immutability (e.g. with a queue subscription that can snapshot the sequence for debugging purposes).