Addition of an Optional Configuration for Field Selection in Document Store Queries

[luigirende]

Feature Description

I propose introducing a new optional configuration for the Query functionality in document-type stores. This configuration would allow specifying a limited number of fields to be extracted directly from the store, reducing the amount of data returned in the response.

Motivation

The current implementation of queries for document-type stores always returns all the fields of the documents in the data store, regardless of the document size or the number of fields actually needed by the application. This approach has several significant inefficiencies:

• Excessive Network Traffic: When entire documents are returned, the volume of data transferred from the data store to the client can be very high, especially when documents contain many fields or large amounts of data. This results in inefficient bandwidth usage and can slow down overall system performance.
• Computational Overhead: Sending complete documents means that the client must process a larger volume of data, even when only a few fields are needed. This overhead can increase the time and computational resources required for unmarshalling and processing the response items at the application level.
• Poor Resource Optimization: Always returning all fields of a document does not allow for optimal resource use, especially in scenarios where large documents are managed, and queries are performed frequently. This approach can reduce overall system efficiency, particularly in high-demand environments or with resource constraints.

These inefficiencies highlight the need for a configuration that allows selecting only the necessary fields for query operations, optimizing both network usage and computational resources.

Implementation Details

In the metadata configuration, the user will be able to explicitly define the desired response structure by specifying the field names and the corresponding JSON paths to retrieve the values from the store.

The proposal has already been partially implemented and covers the states with query functionality enabled, including Redis, MongoDB, CosmosDB, PostgreSQL, and CockroachDB.

Benefits

• Efficiency: Significant reduction in data traffic between the data store and the client, optimizing query operations.
• Flexibility: Developers can more precisely configure the data they want to retrieve, avoiding unnecessary overhead.
• Scalability: This approach is particularly useful in big data scenarios or where large documents are handled.

Conclusion

This feature would improve the efficiency and flexibility of queries in document stores, reducing both network usage and computational overhead at the application level.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[berndverst]

As discussed in my document store proposal I oppose adding more Query functionality to the State Store components.

https://github.com/dapr/dapr/issues/5146

Instead this work should shift to a new document store building block. Perhaps you could lead that effort @luigirende.

The Query API in state store building blocks was a poor design and not be advanced. It will always be Alpha and not receive support.