microsoft / finops-toolkit

Tools and resources to help you adopt and implement FinOps capabilities that automate and extend the Microsoft Cloud.
https://aka.ms/finops/toolkit
MIT License
276 stars 90 forks source link

Power BI: We cannot convert the value null to type Table. #560

Closed robbdi closed 6 months ago

robbdi commented 7 months ago

I recently deployed v0.2 of the FinOps hub but I'm receiving an error in Power BI when I use the Cost Summary and Committed Discounts reports: we cannot convert the value null to type table

Screenshot 2024-01-24 124534

I have verified that the user has the recommended permissions, that the exports are successful and I can see the exports in the storage container, and I see content in both the msexports and the ingestion directories. I've verified that the pipeline successfully ran in Azure Data Factory.

flanakin commented 7 months ago

@robbdi Did you create new FOCUS exports?

FinOps hubs 0.2 requires FOCUS exports and won't work with amortized cost exports. We're updating the docs right now to make that more explicit.

robbdi commented 7 months ago

Screenshot 2024-01-29 081429 As you can see in the attached screenshot, I used the FOCUS dataset for the cost management exports.

thecloudman commented 7 months ago

Same issue here. Used Focus Exports PBI will not work.

An error occurred in the ‘’ query. Expression.Error: We cannot convert the value null to type Table. Details: Value= Type=[Type]

secretsoup commented 7 months ago

Ok I've made some progress on this. I believe this issue is due to this:

https://github.com/Azure/azure-quickstart-templates/issues/13725

Once I modified the export workflow to change amoritizedcost to focuscost I was able to get past the null value to Table error.

However, this brings up a new error, which is this one:

https://github.com/microsoft/finops-toolkit/issues/562

I'm not sure if the out of date quick deploy has other changes to the pipeline that is also affecting this issue but I'm going to deploy the bicep manually again and re-try the entrire thing.

Update:

Spinning it up manually using the bicep still caused me to end up with the Date issue.

flanakin commented 7 months ago

Are y'all also using the Power BI report from 0.2?

There were 2 bugs in 0.2 -- 3 if you count the ongoing AQT issue. Here's a release candidate that fixes them: https://gist.github.com/flanakin/419b5ed778d12a109b297d0dd406c10f I did a quick pass to make sure the files land in ingestion. Will confirm the rest later and then publish the release.

@robbdi FYI - If you open the export details, there's an Export selected dates command that lets you export one month at a time. That might be simpler than creating one-time exports. I'll document that as well.

secretsoup commented 7 months ago

@flanakin Could you advise how to deploy? I get this error:

There was an error downloading the template from URI 'https://gist.github.com/flanakin/419b5ed778d12a109b297d0dd406c10f/raw/bd1bbd8e3a07b09d324ba849bd57a73eca747d14/azuredeploy.json'. Ensure that the template is publicly accessible and that the publisher has enabled CORS policy on the endpoint.

I tried to deploy by copy/pasting the template into a deployment and it just fails.

thecloudman commented 7 months ago

Are y'all also using the Power BI report from 0.2?

There were 2 bugs in 0.2 -- 3 if you count the ongoing AQT issue. Here's a release candidate that fixes them: https://gist.github.com/flanakin/419b5ed778d12a109b297d0dd406c10f I did a quick pass to make sure the files land in ingestion. Will confirm the rest later and then publish the release.

@robbdi FYI - If you open the export details, there's an Export selected dates command that lets you export one month at a time. That might be simpler than creating one-time exports. I'll document that as well.

@flanakin I have been able to deploy the rc version by creating a custom template and pasting the code. I created a one-time export from azure portal directly to mspexports container for one month of data. That worked, the data factory pipelines worked and created the right files in the ingestion container. The problem we have now is Power BI, the ftk_hub function is parsing the folder and looking for a date format in the folder path. The exports dont have this it is a guid:

The error on PBI: An error occurred in the ‘’ query. DataFormat.Error: We couldn't parse the input provided as a Date value. Details: c00fc056-8902-436d-b045-cd28542cd59c01

The details show it is trying to read a folder that does not have a date format.

I suspect the parse folder section of ftk_hub is not able to find the specific folder, may need to adjust the code in PBI to find the right folder. This is my path to the parquet files after an export and import to ingestion: ingestion / UsageDetails / 20240101-20240131/ 202402020214 / c00fc056-8902-436d-b045-cd28542cd59c

flanakin commented 7 months ago

There was an error downloading the template from URI 'https://gist.github.com/flanakin/419b5ed778d12a109b297d0dd406c10f/raw/bd1bbd8e3a07b09d324ba849bd57a73eca747d14/azuredeploy.json'. Ensure that the template is publicly accessible and that the publisher has enabled CORS policy on the endpoint.

@secretsoup Sorry about that! For some reason the portal can't access a gist 😕 I fixed it. Please try again.

flanakin commented 7 months ago

The problem we have now is Power BI, the ftk_hub function is parsing the folder and looking for a date format in the folder path.

@thecloudman Looks like you deployed rc1, which wasn't putting the data in the ingestion folder correctly. Can you deploy the latest?

https://aka.ms/finops/hubs/rc

thecloudman commented 7 months ago

@flanakin attempting to deploy RC2. First deployment failed on storage setting permissions. Doing it again using a new hub name instead of an existing hub name.

thecloudman commented 7 months ago

@flanakin RC2 worked. Connected PBI, all seems okay from a connection point of view. Happy to close out this issue. Legend!

robbdi commented 7 months ago

I was able to deploy RC2 and it seems to correct the issue with the Cost Summary report. I will try the Commitment Discounts report to see if it's working.

flanakin commented 6 months ago

Thanks, all!