(custom) cost price/cost tables/associations

[mmariani]

Hello!

I need to add rating rules to the system metrics. For example, CPU usage would have a price depending on the host type. I see instance prices are mentioned in https://github.com/operator-framework/operator-metering/issues/18

I don't need anything fancy for now, but I am wondering if it's best to extend the existing reports with a set of price (SQL) tables, or write a separate rating component with arbitrarily complex rules. In the latter case, I would derive rating reports from metering reports. It would be harder to get right at first but easier to maintain when adding metrics and rules.

Have you thought of the rating/billing aspect, other than relaying the information from an upstream cloud provider (AWS, Google, Azure)? If you decide to implement something in a near release, I cloud change my mind on how to approach the issue.

Thanks a lot.

[chancez]

This is something we've thought a lot about, and have begun the underpinnings for enabling this. We don't have a lot of documentation for supporting this, but I can give you a rough idea.

Generally the plan is as follows:

• Support referencing arbitrary Presto tables from reportDataSources (done)
  • https://github.com/operator-framework/operator-metering/blob/master/Documentation/reportdatasources.md#prestotable-datasource
• Support referencing arbitrary reportDataSources in reports (done)
  • https://github.com/operator-framework/operator-metering/blob/master/Documentation/reports.md#inputs
• Define a common schema for all billing correlation. Eg cost per cpu, cost per byte of memory, etc.
• Update the "AWS" billing reports to just become "billing reports" that support referencing a "cost association reportDataSource" (the bullet preceding), where this reportDataSource references a table that matches the common schema for cost correlation.
• Redefine the existing AWS billing reportDataSources to expose a table schema matching the "cost associate reportDataSource" schema.

All together this would then mean:

• For each cloud, you would define a prestoTable/reportDataSource that matches a specific schema for unit costs of consumable resources.
• Once our existing billing queries are made agnostic of the cloud; they would then work with "any" reportDataSource/table that matches the schema containing unit costs.
• Define reports and set an input that would allow you specify the name of the specific reportDataSource you wish to use for cost association.

For other clouds, we would probably still need custom code to integrate with each cloud's billing reports, and automate the creation of the PrestoTable and ReportDataSource for each cloud's specifics. However, today you could still create the PrestoTables manually and the reportDataSource's that reference those tables.

Since we're in the middle of working towards in a GA release within openshift this isn't really something we're focused on until that happens, but all of the above totally possible with the functionality we provide today.

[chancez]

Note, that all of this assumes we can define a common schema for how to do cost. This is tricky. In a lot of cases, it may be based on machine type, or you may want to blend the cost across the entire compute cost and take a percentage.

I imagine, we will need different types of "cost schemas" for different use-cases, and our billing reports would need be designed to work with different types of cost schemas.

Eg: we could have a billing report that reports cost entirely based on what machine a pod lands on, and cost could be directly based on the cost of the machine a pod runs on, and another report could do cost by getting the total compute cost of the entire cluster, and do cost based on the % of overall cluster usage each pod consumes.

[chancez]

I updated the title to reflect the terminologies we've been using, but let me know if it doesn't quite match what you were attempting to ask for.

[mmariani]

Thanks for taking the time to explain.

To avoid misunderstandings, I am trying to come up with a generic model for the billing of any metric (system or application) including those that have to relation whatsoever with the underlying billing provided by any cloud provider. An example would be FPGA usage which is measured separately from the usual k8s metrics.

As far as I understand, the roadmap you described will allow us to compute prices inside the reportgenerationqueries.

A price-per-unit model can be implemented with price tables and label values. These prices can even be retrieved from a different datasource, for ease of updating and so on.

On the other hand, if I were to implement a plan with thresholds (i.e. 10$ per month for 100 hours, 0.2 for each hour after that) I would not easily have the required data, like the total hours per tenant (where tenants could be defined in terms of owned namespaces, or some label). When crossing the threshold, the usage must be computed with at least two different prices, which is tricky to express in declarative SQL.

To complicate matters, contracts would not start the same day of the month for all tenants, and afaik the metering operator could process reports in parallel or out-of-order.

Another thing that is deemed important to some, and can easily be done in a separate process, is sending notifications to the tenants (or an orchestrator) as soon as the resources cross a price threshold.

Now if report tables were updatable, I could leave a blank column for the computed price, and fill it from a separate process. Looking at the Hive docs, that would be a bumpy ride. Right now I am leaning towards an external RDBMS. With data already aggregated by hour, the performance should be acceptable.

I presume your remarks about %-splitting the cost of the whole cluster are in the context of chargeback inside a company, which is not my case. That's not a price model that can be easily explained to a customer.

Thanks again and please let me know if I misunderstood something.

[chancez]

I think our existing queries would likely be unable to support the more complex model your describing, as it wouldn't be feasible to support all the use cases in a single set of queries. That being said, you certainly could write new queries that support the models your discussing. Just because SQL is declarative doesn't mean you couldn't easily compute cost for something such as 10$ per month for 100 hours, 0.2 for each hour after that. Part of the idea behind metering is that for more advanced use-cases, you can write your own SQL and have the power to define how you want to do metering.

We don't really have any plans for notifications, you could use the watch API on reports to wait for changes, or you could track the changes to report results. I think this is tricky and is better done by other tooling that consumes the report data potentially. I think the knative/serverless functions approach are good candidates for some of this kind of specific need that can probably be done relatively easily with those frameworks.

Regarding table updates. We simply cannot make tables "updatable", the correct method for that is recreating them from the original source data. The design of the system follows typical data analytics/bigdata pipelines, so all data is effectively immutable once written. Many of our components do not support modifying individual rows, and updates in general are also limited. This may mean creating more reports per time period so that you can limit the actual amount of data in a single report, in case you ever need to recreate them. I think making this a first class feature could make sense in the long run. For example, we could add a ReportTemplate resource which produces multiple reports for configured intervals. This way you could say create a ReportTemplate that automatically creates a report per month or per week, and each of those reports could run either daily/weekly, and their reportingStart/reportingEnd would be templated by the ReportTemplate so that there is a report per period.

You're correct that splitting cost across the clusters whole cost is more of a chargeback model, but it still makes sense if you were to limit the "total cluster cost" to the subset of nodes each customers pods could run on (eg tainted nodes) and produced a report for each tenant/grouping of nodes.

[mmariani]

Just because SQL is declarative doesn't mean you couldn't easily compute cost for something such as 10$ per month for 100 hours, 0.2 for each hour after that.

Not easily. I need to reference and update the current usage totals for the billing period, per each customer, who owns multiple namespaces, and whose billing period can start at any time of the month. When a data frame crosses a usage threshold, I have to split it and compute the total with two prices, and so on. All while locking the tenant, and computing the reports strictly in order from the oldest to the latest, otherwise the results are not reproducible. I'm sure it can get pretty messy given that Presto doesn't have stored procedures.

Regarding table updates. We simply cannot make tables "updatable", the correct method for that is recreating them from the original source data. The design of the system follows typical data analytics/bigdata pipelines, so all data is effectively immutable once written.

Yes, I am well aware of that, and it makes sense to me.

Thanks for the ideas and the prompt reply.