m-kovalsky / fabric_cat_tools

Supercharge your Microsoft Fabric development with the fabric_cat_tools library
MIT License
100 stars 14 forks source link

Unable to migrate the calculated tables into direct lake mode from import mode semantic model #8

Closed hareesh591 closed 1 month ago

hareesh591 commented 1 month ago

Describe the bug

I'm trying to migrate existing import mode semantic model into direct lake mode. We have one calculated table which is created using Calendar() DAX function. While doing the migration , we are able to migrate the calculated table into Lake house but failed to load the same calendar table into direct lake mode semantic model. I'm noting down the reference links below which I followed along with the screenshots. Could you please investigate the issue and provide me the solution ?

I have followed all the steps which are mentioned under below links

https://github.com/m-kovalsky/fabric_cat_tools/blob/main/Model%20Optimization.ipynb https://data-mozart.com/migrate-existing-power-bi-semantic-models-to-direct-lake-a-step-by-step-guide/

![Calculated table error](https://github.com/m-kovalsky/fabric_cat_tools/assets/168816400/60c48825-35f3-42c5-b4d6-eb95b9577670

m-kovalsky commented 1 month ago

I presume this calculated table also has calculated columns. Calculated columns are not supported in the migration, thus this table will not be supported in migration. The Calendar table would be created in your lakehouse but it would only contain the original 'Date' column created and not the additional calculated columns thus it would be unable to be migrated.

hareesh591 commented 1 month ago

HI @m-kovalsky ,

Thank you for your prompt response on my query. Calculated tables will obviously contains the calculated columns which are created using the DAX functions. Do you mean that first I need to create this date table manually in lake house and then bring it up to the direct lake semantic model?

m-kovalsky commented 1 month ago

Calculated tables will not necessarily have calculated columns. In the case of a calendar table then, yes it is highly likely but in other cases it is not necessarily the case. Yes, you will need to create the calendar table in the lakehouse and then add it to the direct lake semantic model.

hareesh591 commented 1 month ago

HI @m-kovalsky

Thank you for confirmation.

m-kovalsky commented 1 month ago

Migrating the calculated columns from a calculated table which uses the CALENDAR function is now possible in 0.4.0.

hareesh591 commented 1 month ago

Thank you for letting me know @m-kovalsky . Is there any script available to fetch the no.of parquet files/ no.of row groups/ row counts from the semantic models in Fabric ? looking for some help here. Please let me know if any script is available.

Thanks

m-kovalsky commented 1 month ago

You can use the get_lakehouse_tables function and specify the optional parameters as True (extended, count_rows). This will tell you if any of your lakehouse tables will hit the guardrails by comparing lakehouse table stats with the guardrails for your capacity's SKU.