The registry is slow to respond when handling a large number of registration requests.

[VenkateswaranJ]

This is related to the https://github.com/AMWA-TV/is-04/issues/17

I have a node with 2500 subresources and when the device boots up it tries to register the entire node to the registry and start heartbeat.

Here is the workflow:

1. Device boot.
2. Find a registry (assume always success).
3. Register the node.
4. Start the heartbeat for the node (let's say 5 seconds).
5. Register the device.
6. Register senders, receivers, sources and flows of the device (requests are sent asynchronously, so within a second we can fire up to 2000 to 3000 requests to the registry)

The issue occurs in stage 6 when the registry is overwhelmed with 2000 to 3000 requests within a second. Each request takes approximately 15 seconds to receive a response. This delay causes the node heartbeat service to fail, as I wait for a response before sending the next heartbeat request. The registry, interpreting the delay as a lack of heartbeat, deletes the node.

Is there any threshold for HTTP requests to the registry?

[garethsb]

As you said, registration requests for Source, Flow, Sender and Receiver do not need to be serialised with respect to each other, although this may complicate conformance with the fail-over specification in the case of errors. Heartbeat requests should be serialised with respect to each other and the initial registration of the Node resource, but should be asynchronous with respect to other registration requests so that delays in registration don't affect heartbeating.

Some questions...

Is that 15 seconds total for 3000 requests?

How many concurrent TCP connections are you using?

Are you keeping connections open to issue multiple requests?

Are you serialising requests with previous responses on each connection or trying to use HTTP/1.1 pipelining?

[VenkateswaranJ]

Is that 15 seconds total for 3000 requests?

No, I just fired 3000 requests and waited for the response asynchronously, but parallel to this I also sent heartbeats for the node and the registry takes 15 seconds to respond to the heartbeat request (this response time increases if I stress the registry with more asynchronous subresource POST update).

How many concurrent TCP connections are you using? Are you keeping connections open to issue multiple requests?

At first, I did everything in a single TCP connection, but I needed to be synchronous to reuse the same TCP connection (sending the next request only after receiving the response for the previous one) but takes like 20 minutes to register the entire node (along with all it's sub-resources), there might be one more reason for this is the registry located far away from the node and not in the local network. Then I moved to the asynchronous approach, but this time I lost the possibility of reusing the single TCP connection for all the HTTP requests. This might be the restriction in the REST library I use. This creates 3000 TCP connections, but surprisingly this brings down the registration time from 20 minutes to 20 seconds, but the only problem is it overloads the registry and increases response time.

Are you serialising requests with previous responses on each connection or trying to use HTTP/1.1 pipelining?

HTTP/1.1 pipelining

[garethsb]

I think you need to look at several aspects of your setup:

• your network, and the roundtrip delay between your node and the registry
• the CPUs you allocate to the nmos-cpp-registry
• best practices for (a) reusing connections for HTTP requests, (b) how many simultaneous connections to open - and how to achieve this with the REST library you use

In my test environments:

1. When running nmos-cpp-registry and a node on the same host so that the networking stack is used but nothing is sent to the wire, 3200 sub-resources can be registered using sequential requests on a single TCP connection (no HTTP pipelining) in about 6-8 seconds depending on the host machine (<2.5 ms per request).
2. When using the nmos-cpp-registry that is available for AMWA members on the AMWA VPN, for which roundtrip delay from me is about 15ms, 3200 sub-resources can be registered in just over 50 seconds (same HTTP/TCP usage as 1). This nmos-cpp-registry is running on a tiny AWS t2.micro instance that has 1 vCPU. It can be seen from the Server-Timing HTTP header reported by the nmos-cpp-registry and measuring at the Node that processing time is still at ~2.5ms per Registration API request and the bulk of the 50 seconds is due to the request/response network delay.
3. Repeating the above test with 4 Nodes all registering 3200 sub-resources simultaneously (therefore 4 parallel connections for registration, and also 4 for heartbeating), there is little difference, all Nodes complete their initial registration in under 55 seconds.
4. Launching 10 same-sized Nodes simultaneously, the performance of the nmos-cpp-registry on the t2.micro instance is affected, I believe due to context switching on its 1 vCPU, and it takes about 250 seconds for all 10 Nodes to complete their initial registration. Averaging processing is now around 50ms, with 1% taking more than 200ms. Network delay is unaffected, still approx. 15ms.
5. Assigning the Registry more CPU cores reinstates the performance with 10 or more Nodes registering simultaneously.

Conclusions:

• the number of simultaneous connections/registrations that can be supported by nmos-cpp-registry depends on the host resources allocated to it, deployments should tune this based on the expected usage
• you may decrease one node's initial time to register by using multiple connections, but it is not a good idea to use 10s or 100s of connections since this may adversely impact the registration of multiple nodes as may happen at network startup, and may be considered a DOS attack by a production server
• you may decrease initial time to make many registration requests on one connection by using HTTP pipelining depending on your implementation's and the registry's support for this (which I believe is not required by the HTTP or IS-04 specifications)
• if you use HTTP pipelining and/or multiple requests, think carefully about which requests can be transmitted in parallel when it comes to updates, and think carefully how you handle 4xx, 5xx and connection errors, since other requests that you have already transmitted may or may not have been processed whether or not you receive a response

[VenkateswaranJ]

Apologise for the delay in response.

• your network, and the roundtrip delay between your node and the registry
• the CPUs you allocate to the nmos-cpp-registry
• best practices for (a) reusing connections for HTTP requests, (b) how many simultaneous connections to open - and how to achieve this with the REST library you use

Currently, the registry is running on 8-core machines alongside a few other services. While I'm uncertain about the Round-Trip Time (RTT), it appears to be higher than that of the local network. My primary concern isn't the network delay, but rather the prolonged response time of the registry when it's overloaded with a significant number of requests. Although I haven't delved into the implementation details, it seems like requests are queued up, with the next one only being handled after the completion of the preceding one.

Anyway, the requirement I have in my hand is to register 8 nodes (each with 3000 subresources) within 5 seconds and I don't think it's quite possible :)

Just for my information, do we have any idea about introducing bulk API in the upcoming IS-04 versions? or this problem is not faced by many of the NMOS users?

@garethsb Please close this issue. I'll find a workaround.

[garethsb]

Since your initial, sequential, registration took 20 minutes or almost 500ms per request, and the processing time in this case should have been 1-2 ms per request (which can be verified by looking at the Server-Timing response header I mentioned), it seems like your network is very slow.

Sending 3000 concurrent requests to a simple registry running on an 8-core CPU is not going to operate well. A production registry, or nmos-cpp-registry behind a gateway, may well rate-limit such concurrent requests as part of DoS protection.

The data structure at the heart of the registry is fairly simple and updates to it are currently serialised (apart from health, updated via heartbeats). It would be wonderful if it could be enhanced to allow concurrent updates while still providing the multiple indices required by IS-04 Query API and maintaining the necessary invariants (e.g. unique creation/update timestamps). Even so, in my experience, 10s or 100s of Nodes performing simultaneous initial registration requests has satisfactory performance and 1000s of Nodes can heartbeat and make registration updates successfully.

If you are interested in leading an AMWA Activity to explore a Bulk Registration API extension, do bring a proposal to the AMWA Incubator! Interested Incubator participants can help explore the problem space and craft a formal proposal for an Activity.

I can't close this issue since sony/nmos-cpp is not my repo, I'm just an opinionated long-time contributor. 🙂

[VenkateswaranJ]

If you are interested in leading an AMWA Activity to explore a Bulk Registration API extension, bring a proposal to the AMWA Incubator! Interested Incubator participants can help explore the problem space and craft a formal proposal for an Activity.

I'll try to implement a simple /bulk API and see if it can help. Because even the bulk messages are chunked after 64kb, I guess it slightly improves the speed of the registration process.

I can't close this issue since sony/nmos-cpp is not my repo, I'm just an opinionated long-time contributor.

:+1:

[garethsb]

I'll try to implement a simple /bulk API and see if it can help.

I did this previously in a fork of nmos-cpp now lost to me. Hardest part is just ensuring we can express simple, consistent, handling of errors that would normally cause a single-resource-registration request to be rejected.

My approach was to allow bulk registration of a single resource type at a time, which therefore could not have ownership interdependencies. I used a request schema like registrationapi-resource-post-request.json but with an array of objects for data.

I used a response schema like IS-05 bulk-response-schema.json.

Another thing to consider is whether such a bulk request can be used for registration updates as well as initial registrations, and what happens if you include data for the same resource more than once in the array. IS-05 Behaviour - Salvo Operation defines this for the Connection API, and I think a similar statement could be sufficient for IS-04 Registration API bulk operations.

Deletion is currently handled by separate DELETE endpoints, and I didn't implement a new bulk delete - the main use case is covered already in that if you delete a Device or Node all the sub-resources are automatically deleted.