Queries regarding consistency in vespa

[107dipan]

I am trying to understand the flow for writes(new and update) in vespa and I have a few queries regarding the flow. Opening a Jira since this is more maintainable.

1. When adding a new document Flow -> a. Distributor calculates the bucket to which the document should be added and dispatches the request to the proper nodes. b. The distributor will dispatch the request to proper content nodes and will wait for the ack to respond to the user. c. On the content node, entry will be added to the transaction log. d. Index fields would be written to the disk, Document written in doc store and attribute fields would be written to memory. e. Attribute fields would be flushed to the disk based on size(or other params). f. In case of node failure, transaction logs would be replayed to figure out the latest changes made at attribute level that have not yet been flushed to the disk.

2. When updating a existing document Flow -> a. Distributor calculates the bucket and dispatches to the proper node. b. Entry will be made to the transaction log. c. For index fields, doc will be read from the doc store, a memory index based on B-tree is created to which this field will be added if not already present after which doc will be written to the document store. d. Memory index is flushed based on size. e. Attributes will be treated similarly. f. In case of node failure, transaction logs would be replayed to figure out the latest changes made at attribute level and index that have not yet been flushed to disk.

Please let me know if my understanding of the flows is correct.

[kkraune]

https://docs.vespa.ai/en/reads-and-writes.html describes the Vespa write path.

1: A distributor dispatches the write to the correct content node(s) yes. The operation is written to attribute memory, but flushing to disk happens later, once enough data has changed. A transaction log is used to persist writes at a high rate, and the document summary store stores a copy of the document. If a node fails, distributors will schedule bucket sync operations across nodes - do not confuse this with the transaction log, which is just a short lived log local to the content node process.

2: More or less the same as 1. Again, flushing is out of band with writes, see https://docs.vespa.ai/en/proton.html#proton-maintenance-jobs

Please feel free to create a pull request to documentation if something is unclear ;-)

[107dipan]

I am unclear about the memory index part that is mentioned on this documentation. - https://docs.vespa.ai/en/partial-updates.html Are the indexes also maintained in memory or only the recently updated indexes are stored in memory?

[bratseth]

Only the recently updated.