Proposal for cube-centric consumption endpoints

[samredai]

The client experiences have naturally become split, having a client type (call it the "reader" persona) for those wanting to pull metadata on nodes or pull sql/data and those who want to contribute towards building the DJ graph (call it the "builder" persona). We've largely thought of reader functionality as a subset of builder functionality, in other words, readers need to view a set of things and builders need to view/edit/create those things.

This is largely correct but I think the CRUDish feeling of the "reader" functionality is not inviting to a user who isn't interested in the DJ internals but just wants to know what data is available for consumption and how can they get it. This is largely what the cube construct is meant to support and I would roughly say that data is available for consumption if:

• A cube is defined that brings a set of compatible metrics and dimensions together
• That cube can return data with very low latency (it's either materialized to a performant datastore or generates a query on the underlying source tables that's already very performant)
• The data is fresh with the acceptable threshold being controlled by the max-stale value in the cache control headers

This proposal is to extend the reader functionality by extending the client and adding some facilitating endpoints to the backend API that extends the cube router to provide a rich "cube only" set of features.

A few examples would be:

• GET /cubes/ - List available cubes with all sorts of options to filter on cube attributes like freshness, materialization types, or even providing a set of metrics/dimensions to only see cubes that contain them as a subset
• GET /cubes/{cube_name}/data/ - A cube focused data endpoint that honors all cache control headers
• POST /cubes/utilization/ - A callback endpoint to report utilization of a cube. This is a flexible endpoint that viz integrations can optionally call to self-report usage of a cube
• ...many more

[agorajek]

@samredai I don't think that anything you said here is controversial, but I may be missing something here... are you saying that this cube-centric functionality should be added to the Reader side but not-added to the Builder side? I think that the ability for the Builder class to inherit all of the Reader class is natural and does not seems like a problem to me. As I said I think I am missing the point here.

[samredai]

@agorajek using the client interface was probably a bit of a distraction but the core of what I'm proposing is that we extend the api by adding a number of /cubes/{functionality}/ endpoints.

We do this today for metrics and dimensions, i.e. the /metrics/common/dimensions/ and /dimensions/common/metrics/ endpoints so I didn't expect it to be too controversial to do something similar for cubes. I just wanted to get everyone's thoughts. I'll start on a PR adding a few of this to give something more concrete to review.

[agorajek]

Ah... yes indeed. Agree on both points: client was a distraction on the topic and the /cubes/* endpoints makes total sense.

[shangyian]

@samredai I agree that extending these API endpoints with more cube-specific ones make sense! There's also already /cubes/dimensions/sql and /cubes/dimensions/data. Once we have better logic to handle the max-stale headers, it'll be much clearer to users what kind of data they're getting back from a cube.

[samredai]

Awesome I'll get started on a couple of these towards the end of this week!