databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

uc-04-system-tables - missing dbu in system.usage.billing tbl #55

Open evogelpohl opened 11 months ago

evogelpohl commented 11 months ago

Not directly related to this demo, but found using it.

I have enabled the Security & Compliance Profile feature on a workspace in Azure. I'm seeing a new DBU in my Azure Consumption export (the azure mgmt api for consumption):

"meterCategory": "Azure Databricks", "meterId": "1bd9932a-deed-51f3-82bc-fd87db26fc07", "meterName": "Premium Enhanced Security and Compliance DBU", "meterRegion": "US East", "meterSubCategory": "Azure Databricks", "previousInvoiceId": "", "pricingCurrencyCode": "USD", "product": "Azure Databricks - Premium - Enhanced Security and Compliance - US East",

at an effectivePrice = 0.1 for my pay-as-you-go plan.

This SKU is not in select distinct(sku_name) from system.billing.usage. It's not reflected in this demo's helper function that makes the pricing lookup table either.

Thus, our true cost for Azure Databricks as seen in the Azure Cost Portal doesn't tie out to the usage log.