Evaluate capability and performance of using Trino directly as exchange microservice caching layer, or identify a dedicated cache to do so

os-climate / os_c_data_commons

Repository for Data Commons platform architecture overview, as well as developer and user documentation

Apache License 2.0

21 stars 10 forks source link

Evaluate capability and performance of using Trino directly as exchange microservice caching layer, or identify a dedicated cache to do so #263

Open caldeirav opened 1 year ago

caldeirav commented 1 year ago

The data exchange layer will have microservices querying data repeatedly. Can we simplify the architecture by plugging microservices directly into Trino with materialized views providing the caching, or do we need an underlying cache service such as Redis? Or can we achieve this by leveraging only Trino materialized views and what would be the performance comparison?

https://trino.io/docs/current/connector/redis.html

caldeirav commented 1 year ago

Alternative used by Netflix is to have a GraphQL capability on top of Trino: https://netflixtechblog.com/how-netflix-scales-its-api-with-graphql-federation-part-1-ae3557c187e2 https://netflixtechblog.com/data-movement-in-netflix-studio-via-data-mesh-3fddcceb1059