FgForrest / evitaDB

evitaDB is a specialized database with an easy-to-use API for e-commerce systems. It is a low-latency NoSQL in-memory engine that handles all the complex tasks that e-commerce systems have to deal with on a daily basis. evitaDB is expected to act as a fast secondary lookup/search index used by front stores.
https://evitadb.io
Other
61 stars 7 forks source link

Provide means for accessing reference count without actually fetching the entities #650

Open novoj opened 1 month ago

novoj commented 1 month ago

After recent discussion with FE team using evitaDB, there may be use case were we want to fetch entity that has existing referenced entities of some type. On top of that however, we want to fetch count of these referenced entities without actually fetching the entities. Something like that:

{
  listGroup(
    filterBy: {
      attributeCodeEquals: "news-group"
      referenceProductsHaving: {}
    }
  ) {
    productsCount # generated from reference `products`
  }
}

I think it could be solved also with facet summary, probably like this:

{
  queryGroup(
    filterBy: {
      attributeCodeEquals: "news-group"
      referenceProductsHaving: {}
    }
  ) {
    extraResults {
      facetSummary {
        products {
          count
        }
      }
    }
  }
}

But that's quite cumbersome to use on FE. @novoj do you think it would be valid to support the first approach at the GraphQL API level? We could also reuse the filterBy clause from the reference fields. On the backend it could be translated to basic referenceContent.


I think we could automatically provide the count attribute on EntityDecorator level. evitaDB internally always needs to compute reference primary keys to calculate the count, so we could always provide the array of these primary keys (since they have to be fetched anyway). But on the API level the primary keys can easily be thrown away and only the count could be provided to the external clients, thus saving some transport/networking costs.

novoj commented 1 month ago

This feature request partially opens up a path to a larger extension of the reference fetching enhancements. It would be also beneficial to be able to fetch only chunks of references. After discussion with @lukashornych we'd like to:

  1. be able to avoid sending all the referenced PKs over the wire in all of the protocols (so it should be somehow reflected in the base query language)
  2. be able to paginate the fetched references
  3. combine both above requirements with ordering and filtering constraints

In order to do so, this language extension makes sense to us:

// returns only counts for product reference (no PKs or bodies are sent)
referenceCount('products')
// returns only counts for all entity references (no PKs or bodies are sent)
referenceCount()
// returns only counts for product reference that match filtering criteria in a custom order (no PKs or bodies are sent)
referenceCount('products', filterBy(...), orderBy(...))
// returns first page of 20 primary keys that match filtering criteria in a custom order
referenceContent('products', filterBy(...), orderBy(...), page(1, 20))
// returns first page of 20 referenced entities that match filtering criteria in a custom order
referenceContent('products', filterBy(...), orderBy(...), entityFetchAll(), page(1, 20))

We still need to analyze how this proposal affects the internal structure of EntityDecorator and the connected APIs. This change would also affect all the APIs at once and would cover all our requirements.