Monitoring support

[rsbh]

We are running multiple cube instances in our cloud and want to monitor them like other services. Can we add support for statsd/new-relic or some similar services?

[paveltiunov]

Hey @rsbh ! Thanks for posting it! Could you please elaborate what kind of metrics do you want to track?

[rsbh]

Hey @paveltiunov, I am looking for metrics like resource (IO, CPU, Memory) consumption, API throughout and latency etc.

[paveltiunov]

@rsbh I guess you're looking for a standard monitoring agent like Newrelic or Datadog when you deploy it in docker. No additional changes are required from Cube.js side.

[rsbh]

I think for new-relic, we need to add its JS sdk to cube.js. https://docs.newrelic.com/docs/agents/nodejs-agent/installation-configuration/install-nodejs-agent/

[rchkv]

Hey @rsbh 👋

Is the issue still actually? Do you need any further assistance? Feel free to follow up 🙂 If the issue doesn't actually, could you please close it? Thanks!

[rsbh]

Yes, i haven't able to get monitoring setup on cube.js instance. Can you guys recommend some way to get metrics from the server ?

[rchkv]

@paveltiunov, could you please recommend something?

[soniafaria]

Hi, I was too trying to implement monitoring to my cube.js, I tried to use prom-client, allowing prometheus to scrape some metrics. But didn't succeed installing prom-client to cube.js, is there any impediment to install external dependencies in cube.js as in any other node.js application? If there is, is there any other way to allow prometheus to scrape metrics in cubejs? Thank you.

[rongfengliang]

@rsbh @soniafaria I write one simple demo with express Prometheus middleware https://github.com/rongfengliang/cube.js-prometheus may be useful

[kapilt]

it would be nice to have some builtin in metrics exposed by cubejs components, ala prometheus or open telemetry support, rather than having to do direct code integrations/modifications to get observability.

[paveltiunov]

Please see https://cube.dev/cloud/ for native Cube monitoring support.

[ChrisLahaye]

I believe OP is using self-hosted Cube in their cloud, not Cube's cloud.

The industry standard for monitoring is Prometheus. It would be nice to have some Prometheus metrics exposed by Cube.js and Cubestore to monitor health and performance, e.g. number of queries, query duration, slow query counter, etc.

@paveltiunov Could we please reopen this feature request for monitoring and observability in the server itself?

[paveltiunov]

@ChrisLahaye Cube Platform is available on-prem as well. Integration with Prometheus in Cube Cloud will arrive this quarter.

For OSS, I guess you can write an addon that collects logs from Cube and Cube Store, processes them, and sends them to Prometheus. It's how it's done in Cube Cloud. Separate GitHub repo would be ideal for this type of contribution. We can then link it in our docs as third-party monitoring implementations.

[github-actions[bot]]

If you are interested in working on this issue, please leave a comment below and we will be happy to assign the issue to you. If this is the first time you are contributing a Pull Request to Cube.js, please check our contribution guidelines. You can also post any questions while contributing in the #contributors channel in the Cube.js Slack.

[ChrisLahaye]

@paveltiunov I think writing logs and analyzing those to compute your metrics introduces a lot of unnecessary overhead. You will have to stay in-sync with the log messages and formats used in the main project and potentially apply parsing and pattern matching.

A better solution would be to add dedicated but optional Prometheus instrumentation (prom-client) in the main project so metrics can be easily added and maintained in core spots of the project and then expose them using a separately configurable server (as Prometheus is intended to be pull-based).

What do you think about this proposal? Would you potentially accept a PR for this?

[paveltiunov]

@ChrisLahaye Thanks for such detailed feedback! The thing is for most Cube metrics and traces you'd need to aggregate and analyze logs from multiple Cube API nodes and refresh workers. So you'd need to analyze data from the whole Cube cluster in order to get any insights. You also would need to aggregate those and join together to analyze query trace spans due to idempotency of query queue. So it's definitely standalone service / server which should do this. In terms of API logs seems to me quite stable way of analyzing it as it's what used in Cube Cloud. I don't think we need to introduce and maintain bunch of different APIs here.

Merging and maintaining monitoring support is unfortunately outside of core team capacity. Every contributed code piece means core team need at least review fixes and enhancements for this code piece. We're thinking about establishing code ownership for community supported packages so we can have community supported packages in main repo that don't require core team time commitment. We're going to do pilot with existing ones before we move on to adding more major ones like monitoring.

[ChrisLahaye]

@paveltiunov

aggregate and analyze logs from multiple Cube API nodes and refresh workers. So you'd need to analyze data from the whole Cube cluster in order to get any insights."

I understand that. Prometheus is perfectly capable and successfully being used in distributed architectures as well. In Prometheus you can define jobs that define the targets whose metrics you want to scrape. These targets can be discovered by various service discovery methods. Metrics are automatically labeled with target properties like the namespace, container, instance, etc.

When querying the metric you specify the aggregation, so you can for example query the average or the 90th percentile query duration of all refresh workers.

So it's definitely standalone service / server which should do this. In terms of API logs seems to me quite stable way of analyzing it as it's what used in Cube Cloud. I don't think we need to introduce and maintain bunch of different APIs here."

It would help to have at least one solution available in the open-source project as well, as currently there are no means for simple monitoring and observability.

Merging and maintaining monitoring support is unfortunately outside of core team capacity. Every contributed code piece means core team need at least review fixes and enhancements for this code piece."

This is not really encouraging to contribute to the project. Well, better to know before hand, than to put in effort and have the PR rejected. It feels like you want to focus on a solution that is only available in Cube Cloud, instead of integrating the de facto standard for monitoring and observability in the open-source product. I understand that this is ultimately your decision, but I humbly ask you reconsider and to research how Prometheus can assist in your current goals and how it will benefit the projects and the community.

[rongfengliang]

@ChrisLahaye I think you can still use express Prometheus middleware. and we can easily integrate any express Prometheus middleware (express-prom-bundle,@apio/express-prometheus,express-prometheus-middleware). and we can also do some extensions for cube.js (https://cube.dev/docs/config)

[ChrisLahaye]

I am just gonna put it here since I see this PR being mentioned. Simply adding a Prometheus middleware for express is not sufficient to provide meaningful monitoring and observability. It only provides generic metrics about the specific Node process, requests and responses. It won't provide any information about the internals like caching and queuing.

The suggestion to create metrics based on log analysis is a bad work-around. It is inefficient to output logs and even worse to parse them. It also saturates the output with useless logs just for the purpose of creating metrics. It is also hard to maintain due to the dependency on the log format.

Prometheus is battle-tested and makes it very easy to add, maintain, and export metrics. The Node library is properly maintained and only has a single dependency. There is even a Rust library which could be used for Cubestore.

@ChrisLahaye Cube Platform is available on-prem as well. Integration with Prometheus in Cube Cloud will arrive this quarter.

For OSS, I guess you can write an addon that collects logs from Cube and Cube Store, processes them, and sends them to Prometheus. It's how it's done in Cube Cloud. Separate GitHub repo would be ideal for this type of contribution. We can then link it in our docs as third-party monitoring implementations.

Please reconsider, monitoring and observability is a necessity for production systems. Not only in Cube Cloud, but also in Cube OSS.

[paveltiunov]

@ChrisLahaye It might be we're thinking about very different levels of monitoring and sets of metrics. I'd suggest building something and seeing how it works. For those who want to take a stub on monitoring support, please create an npm package that uses logs either instruments Cube. We will expose it in our docs first and most likely will support API it uses to instrument Cube.