Add a new Streaming Query Records endpoint to support logical store with large number of records

[ghost]

What happened: When we query records in store that have lots of entities or for records with large opaque string, the response often goes over the maximum gRPC receiving message size set on the gRPC client. When this happens, the client either gets an error because the payload goes over the max response message size or the calls times out because there are too many records to return within the read timeout value.

What you expected to happen: I'd like to introduce a new StreamingQueryRecords endpoint to return a stream of records instead of a unary gRPC message to not be constraint by the limit set on the client. The operation will support an offset parameter, which will be handle transparently in Open Saves to minimize access cost based on the offset value following the pseudo-code below: (1) Support for an offset and limit parameter (2) gRPC streaming response for all results (3) When offset == 0, perform the query and stream the results as is (4) When offset > 0, first query the entities with keysOnly=true, then fetch records using a new metadb.GetMulti function that will be introduced with these changes

How to reproduce it (as minimally and precisely as possible): Create 5 records with each <~1MB in their opaque string. Even though the number of records is small, the query endpoint fails.

Anything else we need to know?:

Environment:

• Open Saves version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Install method (yaml/helm):
• Troubleshooting guide log(s):
• Others:

[hongalex]

Sounds good to me. The only thing I would suggest is just exploring is whether or not we have to completely remove QueryRecords or if we can just add an additional RPC called StreamingQueryRecord which can be used for larger responses like this.

[ghost]

I reviewed the interface again and it makes sense to keep the original QueryRecords function. I will modify the original issue to describe a new interface instead of changing the existing one. Below is some rational:

By default, a Golang gRPC server has no limit on the message it sends back to a client (defaultServerMaxSendMessageSize = math.MaxInt32), only when receiving messages from a client (4MB). The issue we see is most likely caused by a limit set on the gRPC client, which triggers an error when the message size is over 4MB.

The DialOption can be changed to accept a different maximum with WithDefaultCallOptions(MaxCallRecvMsgSize(s)). I am still trying to confirm that the default is 4MB. I'll update these comments when I find out for sure.

[ghost]

I confirm that increasing the limit in the gRPC client provides a work around for payload that are higher than 4MB. However, there is going to always be a limit that depends on what the client sets.

[ghost]

After discussing this with Alex, we have decided to not introduce a streaming endpoint for querying records. The client can paginate using both offset and limit offered in the new implementation. For clients that are expecting lots of data in the records, they can bump up the default maximum receiving gRPC bugger size to > 4MB.