search

opencog / atomspace

The OpenCog (hyper-)graph database and graph rewriting system
https://wiki.opencog.org/w/AtomSpace
Other
795 stars 225 forks source link

Implement `WriteBufferProxy` #3070

Closed linas closed 1 month ago

linas commented 1 month ago

Enhancement: Implement a WriteBufferProxy. This would be just like the other ProxyNodes except that it would collect up a bunch of write requests, before forwarding to the next stage. This is needed to avoid "write amplification".

Current pair-counting code increments a count, and then immediately calls store-atom (which, for RocksStorageNode writes to disk.) Counting proceeds at hundreds of thousands per second, so this does vast amounts of disk writes. Buffering would reduce this.

The simplest buffering design is to place even new write request (i.e. handle) into a std::set, so that duplicate entries are automatically eliminated. Then this set would be flushed every N seconds. Fancier designs are also possible.

Even easier would be to use https://github.com/opencog/cogutils/tree/master//opencog/util/async_buffer.h which already does the needed locking and bufering, it just needs to be wrapped in the proxy-node gorp.

linas commented 1 month ago

Other kinds of proxies that build on this are described in https://wiki.opencog.org/w/ProxyNode#List_of_ideas_for_other_Proxies

linas commented 1 month ago

I'm working on this now.

linas commented 1 month ago

Mostly done.

linas commented 1 month ago

Done in #3071 -- I realized that this was urgent, and it was easy, so I just went and did it. Didn't take that long.