Deploying application with many document types results in HTTP status code: 504

[paolodedios]

Deploying an application using Vespa 6.278.1 with 1000 search definitions on a 5 node cluster consisting of 1 config server, 2 search nodes and 2 content nodes (each with 16GB of RAM and SSD storage) results in HTTP status code: 504. Time 'vespa-deploy' timeout used was 600 seconds but the command returns within 2 minutes with the reported error.

Uploading application '/sample-apps/basic-search-java-noderepo-scale/target/application.zip' using http://localhost:19071/application/v2/tenant/default/session?name=application.zip
Session 3 for tenant 'default' created.
Preparing session 3 using http://localhost:19071/application/v2/tenant/default/session/3/prepared?timeout=600
Request failed. HTTP status code: 504
Empty response
real    2m10.618s
user    0m0.083s
sys     0m0.015s

All search definitions were deployed to a single content cluster with a services.xml configuration similar to the following:

 <content id="mydocs" version="1.0">
        <redundancy>2</redundancy>
        <documents>
            <document type="mydocs_1" mode="index" />
            <document type="mydocs_2" mode="index" />
            <document type="mydocs_3" mode="index" />
            <document type="mydocs_4" mode="index" />
            <document type="mydocs_5" mode="index" />
            ...
            <document type="mydocs_1000" mode="index" />
       </documents>
</content>

Reducing the number of search definitions to 750 document types results in a successful deploy so it seems like the deploy process is somewhat resource bound on my cluster when the number of searchdefinitions approach 1000 unique document types.

When the application did deploy with fewer searchdefinitions (~750 in my case), it seems to spend a lot of time (hours) creating and deleting index files and exhausting RAM while doing so.

[2018-08-26 21:27:15.609] INFO    : searchnode       proton.proton.server.proton        Add document database: doctypename(mydocs_653), configid(mydocs/search/cluster.mydocs/mydocs_653)
[2018-08-26 21:27:16.212] INFO    : searchnode       proton.proton.server.proton        Add document database: doctypename(mydocs_654), configid(mydocs/search/cluster.mydocs/mydocs_654)

[2018-08-26 21:26:55.117] WARNING : searchnode       proton.searchlib.docstore.logdatastore     Removing dangling file '/opt/vespa/var/db/vespa/search/cluster.mydocs/n0/documents/mydocs_619/0.ready/summary/1535318549346125000.dat'
[2018-08-26 21:26:55.437] INFO    : searchnode       proton.proton.server.proton        Add document database: doctypename(mydocs_620), configid(mydocs/search/cluster.mydocs/mydocs_620)
[2018-08-26 21:26:55.450] WARNING : searchnode       proton.searchlib.docstore.logdatastore     We detected an empty idx file for part '/opt/vespa/var/db/vespa/search/cluster.mydocs/n0/documents/mydocs_62/1.removed/summary/1535318549979589000'. Erasing it.
[2018-08-26 21:26:55.450] WARNING : searchnode       proton.searchlib.docstore.logdatastore     We detected an empty idx file for part '/opt/vespa/var/db/vespa/search/cluster.mydocs/n0/documents/mydocs_62/0.ready/summary/1535318549977902000'. Erasing it.
[2018-08-26 21:26:55.450] WARNING : searchnode       proton.searchlib.docstore.logdatastore     Removing dangling file '/opt/vespa/var/db/vespa/search/cluster.mydocs/n0/documents/mydocs_62/0.ready/summary/1535318549977902000.dat'

[2018-08-26 21:26:55.736] WARNING : searchnode       proton.searchlib.docstore.logdatastore     Removing dangling file '/opt/vespa/var/db/vespa/search/cluster.mydocs/n0/documents/mydocs_620/2.notready/summary/1535318550686543000.dat'

[2018-08-26 21:34:14.517] INFO    : searchnode       proton.vespalib.util.process_memory_stats  create(): Memory stats have changed, trying to read smaps file again: i=5, prevStats={_mapped_virt=237858816, _mapped_rss=25104384, _anonymous_virt=14368985088, _anonymous_rss=8488566784, _mappings_count=17235}, currStats={_mapped_virt=237858816, _mapped_rss=25104384, _anonymous_virt=14384750592, _anonymous_rss=8503005184, _mappings_count=17254}
[2018-08-26 21:34:15.516] INFO    : searchnode       proton.vespalib.util.process_memory_stats  create(): Memory stats have changed, trying to read smaps file again: i=8, prevStats={_mapped_virt=237858816, _mapped_rss=27189248, _anonymous_virt=14422712320, _anonymous_rss=8522080256, _mappings_count=17292}, currStats={_mapped_virt=237858816, _mapped_rss=27189248, _anonymous_virt=14446727168, _anonymous_rss=8536281088, _mappings_count=17307}

The vespa search sizing guide doesn't seem to detail the relationship between the number of unique searchdefinitions/documents to the amount of resources required for a cluster. As a result I have the following questions.

1. I've been launching the 'vespa-deploy' command with a timeout of 600 seconds. It will return a 504 error on the client side without an error on the configserver side some of the time. Other times it will return the following timeout error that is well below the 600 second timeout limit used in the client.

[2018-08-25 00:28:43.278] INFO    : configserver     Container.com.yahoo.vespa.config.server.http.HttpErrorResponse     Returning response with response code 500, error-code:INTERNAL_SERVER_ERROR, message=prepare timed out [112 ms, 181510 ms, total: 181622 ms] after build models step: default.default

Is the vespa-deploy timeout parameter specific to a different part of the deployment process?

2. For a given size search/content node, there seems to be an upper bound to the number of searchdefinitions/documents that can be managed by a single content node. Is there a sizing guide for this aspect of an application?
3. Is there a better strategy for managing thousands of unique searchdefinitions/document types than placing them all in a single content cluster? Should they be manually partitioned to multiple content clusters (in a one document per content cluster or few documents per content cluster strategy)?
4. The system spends a lot of time creating and deleting index files and churning through RAM after deploying a many document-type application. Communication between the search and content nodes start timing out. Is there a way to control the deploy process such that it doesn't consume the entire system's resources?

[baldersheim]

1000 is around 10x higher than what I am aware has ever been tested. I think the fixed cost per searchdefinition is too high to scale above approx 100. I think the main fixed resource is memory. You would definitely also need SSD to handle the I/O as the data are stored in separate directories/files.

[bratseth]

Yes, Vespa is definitely not tuned for this - the search definitions are supposed to correspond to actual data types in the application, which tend to be below a few tens at most. What are you trying to achieve here? Perhaps we can help you find a better way to do it.

[paolodedios]

I am trying to see if it is possible to port an application that accepts and indexes an uncapped number of user-defined document types. More importantly:

1. Each document/schema type needs to be able to be independently managed, fed, and re-fed without affecting other document indexes
2. Each document type needs to have both fixed fields and fields that can be dynamically added and removed from the schema by the user.

It sounds like partitioning and delegating subsets of document types to multiple indexed content clusters would be more manageable for Vespa from a resource management point of view.

Alternatively, can a single search definition be partitioned into groups of related fields that are managed separately as if they were separate document types? Would this strategy allow for feeding/re-feeding of only portions of the data without having to rebuild the entire index?

[bratseth]

Ok, Vespa does have a multitenancy mode, but on the application, not search definition level - most applications do not consist of just a search definition and data but also query profiles ml models, Java components, a particular layout of clusters and data etc., and you typically want isolation between applications for security and availability (avoid one application feeding all others to death etc.). We don't have user level documentation for it, but some people have been walking down this road already, see https://github.com/vespa-engine/vespa/issues/5507 if you are interested.

To your questions, it wouldn't help deployment time to split into multiple content clusters as all of them would still need to be processed on deployment. You would have to split into multiple applications once you go beyond a few hundred.

Alternatively, can a single search definition be partitioned into groups of related fields that are managed separately as if they were separate document types?

Yes, document types support multiple inheritance so you can create a different one for each group of fields then inherit the groups you want into a particular concrete document type.

Would this strategy allow for feeding/re-feeding of only portions of the data without having to rebuild the entire index?

That is supported regardless of how the fields are defined: You can send write operations to any one field, or a set of fields and they will be consumed without impacting the other fields of those documents.

[paolodedios]

Would deploying multiple applications via Vespa's multi-tenancy feature yield benefits over deploying multiple, single tenant clusters through a cluster manager like Kubernetes or Mesos? Does the Vespa multi-tenancy feature provide more granular control over resource management or enforce some sort of SLA per application?

I will experiment with the various deployment strategies outlined above. Thank you for taking the time to answer my questions.

[bratseth]

With the multi-tenancy feature applications don't control what hosts they are allocated - ho hosts.xml and instead of node lists in services you just say . To set this up you set up a node repoitory to which you can add nodes for allocation through a web service (or, if you want to plug in k8s, you can write a plugin which gets nodes on demand).

With that solution Vespa enforces the placement constraints etc. that is needs. They could probably be expressed in k8s at this point, but that was not the case when we did this work. We don't have auto SLA enforcing (e.g autoscaling) yet, it's on the roadmap, but challenging for stateful systems.

[paolodedios]

Thank you for answering my questions. Closing this issue.