Time interval usage metrics by logs from UCAN Stream

[vasco-santos]

Context

Within w3up infrastructure, through the UCAN stream we track system wide metrics and space metrics. These metrics allow us to know overall system metrics and w3up users to know about their total usage. However, we have no visibility on real time volume of usage by each space. Based on the operation of older APIs, we see value on knowing usage volume, so that we can proactively avoid abused and get to know patterns.

Requirements

• Ability to visualise real time usage for the spaces with most usage (ideally in Grafana)
  • ability to filter by capability executed, and if receipt or workflow
• Manual query for usage of specific
• Data durability limited (30 days?)

[vasco-santos]

Available Options

1. Kinesis Data Analytics for Apache Flink + Amazon timestream
2. Kinesis Data Firehose + S3 + Athena

1. Kinesis Data Analytics for Apache Flink + Amazon timestream

• Kinesis Data Analytics for Apache Flink

  • One consumer for the data stream that preprocesses and ingests data into Timestream.
  • Needs Apache flink application with tables

• Amazon Timestream

  • scalable serverless time series database for operational events ingestion and querying
  • allows you to configure different retention periods for your tables to optimize storage costs
  • There is a grafana plugin

References

• https://docs.aws.amazon.com/timestream/latest/developerguide/ApacheFlink.html
• https://github.com/aws-samples/amazon-kinesis-timestream-grafanahttps://docs.aws.amazon.com/kinesis/index.html#amazon-kinesis-data-analytics
• https://aws.amazon.com/blogs/database/near-real-time-processing-with-amazon-kinesis-amazon-timestream-and-grafana/

2. Kinesis Data Firehose + S3 Data Lake + Athena

• Kinesis Data Firehose
  • fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3)
  • no need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified
  • also possible to configure Kinesis Data Firehose to transform data before delivering it
• Amazon Athena
  • Provides a simplified, flexible way to analyze petabytes of data where it lives. Analyze data or build applications from an Amazon Simple Storage Service (S3) data lake
  • Can be integrated with Grafana
    • https://grafana.com/grafana/plugins/grafana-athena-datasource/
    • https://docs.aws.amazon.com/grafana/latest/userguide/Athena-using-the-data-source.html

References

• https://aws.amazon.com/athena/
• https://gargeebhatnagar.medium.com/amazon-kinesis-data-firehose-and-amazon-athena-how-to-collect-the-log-data-using-kinesis-agent-f4beb705bd7fhttps://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html
• (can also see https://aws.amazon.com/blogs/database/near-real-time-processing-with-amazon-kinesis-amazon-timestream-and-grafana/) for Data Firehose