Closed nehajatav closed 2 years ago
Please refer to https://docs.vespa.ai/en/performance/ for an understanding of memory use. This is a large, complex problem, and it is generally hard to sum up such metrics for the rss total.
Commercial offerings like https://cloud.vespa.ai/en/memory-visualizer lets you more easily analyse memory usage per schema / data - alternatively, use the metrics, see https://docs.vespa.ai/en/reference/metrics.html
Undoing fieldset changes and removing fast-search on unused field (no documents has this field populated) brought down the memory consumption. Any idea if one of these two are memory heavy?
@geirst correct me if I am wrong
Adding/removing a fieldset should not have impact on neither disk nor memory footprint.
fast-search builds a an index structure to improve query performance. This index structure will point to posting lists, that will be empty for unused fields. I hence think adding fast-search increases memory usage, even for empty fields. https://docs.vespa.ai/en/attributes.html#index-structures
Changing attribute settings normally requires restarts, see https://docs.vespa.ai/en/reference/schema-reference.html#modifying-schemas - so changes is memory footprint can also be a result of this (structures are flushed, transaction log etc).
Yes, @kkraune you are right. fast-search leads to a slight memory increase even for empty fields due to the extra index structures created. E.g. single value fields will have the "undefined value" when empty, and there is a posting list for this value.
single value fields will have the "undefined value" when empty, and there is a posting list for this value
That explains why in another attempt, we merged the schema for 28 document types into a single document type and memory footprint grew in 1/4th the number of documents would in 28 schema scenario.
Describe the bug Summing up allocated_bytes for all schemas on a given content node is much lesser than memory_rss. What else occupies the same? How can we bring down memory consumption?
To Reproduce Steps to reproduce the behavior:
Expected behavior allocated_bytes across index, attributes, document_store should add up to memory_rss
Screenshots top command from within pod:
metrics/v2/values
state/v1/metrics summing up all the memory.allocated_bytes (converted to GBs across schemas), I don't see more than 11GB consumed on a content node:
Environment (please complete the following information): OS: Docker image vespa:7.559.12 Infrastructure: Kubernetes Versions Major:"1", Minor:"21", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"
Host and services.xml: same as in this comment: https://github.com/vespa-engine/vespa/issues/22315#issuecomment-1113074776
Vespa version 7.559.12 compiled with go1.16.13 on linux/amd64
Additional context The memory footprint used to be much lesser. The only schema changes made after that are (i) adding fieldset (ii) adding fast-search to fields that are not being hydrated as part of feeding client with no allocated_bytes for this field as per state/v1/metrics (iii) converting one field from string to array with overall memory footprint of 130Mb at field level as per state/v1/metrics