search

m-kovalsky / fabric_cat_tools

Supercharge your Microsoft Fabric development with the fabric_cat_tools library
MIT License
110 stars 15 forks source link

migrate_tables_columns_to_semantic_model - source- and target semantic models in different workspaces #2

Closed MarcAHunt closed 7 months ago

MarcAHunt commented 7 months ago

Hi m-kovalsky,

first of all thx for this great tool.

I have a small issue with migrating tables and columns to another semantic model.

It all works fine if source- and target-semantic-model reside in the same workspace, but I can not seem to get it to work if my target-semantic-model is in a different workspace.

What I actually would want to achieve is building new semantic models in a dev-workspace, once I am fine with it, push them to a production-workspace. Your tool indicates this could be possible.

Here is what I'm doing (copied of notebook cell): import fabric_cat_tools as fct import time

newDatasetName = 'targetSM' #Enter the new Direct Lake semantic model name """ comment -> I can create the newDataset in any workspace I want. """ fct.create_blank_semantic_model(newDatasetName, workspaceName='DevWS')

time.sleep(5)

""" comment -> re-creates all tables and columns of source_semantic_model (=param 1) in target_semantic_model (=param 2) -> lakehouseName and lakehouseWorkspaceName define the workspace and lakehouse where the data should be coming from. (Switching datasource, all fine!)

-> PROBLEM: It does not seem to work if source- and target-semantic-model do not reside in the same workspace! """ fct.migrate_tables_columns_to_semantic_model( 'sourceSM', 'targeSM', workspaceName= 'DevWS', lakehouseName= 'LH_Gold', lakehouseWorkspaceName='ProdWS')

fct.migrate_model_objects_to_semantic_model('sourceSM', 'targetSM', workspaceName= 'DevWS')

I am not sure if that's an issue. Could be a feature.

Thx a million for a feedback.

Best Regards from Germany, Marc

m-kovalsky commented 7 months ago

Yes, I am aware of this current limitation. This capability will be added in a future version (you will have another parameter to set the workspace for the new semantic model). Stay tuned...!

From: MarcAHunt @.> Sent: Wednesday, April 10, 2024 4:05:36 PM To: m-kovalsky/fabric_cat_tools @.> Cc: Subscribed @.***> Subject: [m-kovalsky/fabric_cat_tools] migrate_tables_columns_to_semantic_model - source- and target semantic models in different workspaces (Issue #2)

Hi m-kovalsky,

first of all thx for this great tool.

I have a small issue with migrating tables and columns to another semantic model.

It all works fine if source- and target-semantic-model reside in the same workspace, but I can not seem to get it to work if my target-semantic-model is in a different workspace.

What I actually would want to achieve is building new semantic models in a dev-workspace, once I am fine with it, push them to a production-workspace. Your tool indicates this could be possible.

Here is what I'm doing (copied of notebook cell): import fabric_cat_tools as fct import time

newDatasetName = 'targetSM' #Enter the new Direct Lake semantic model name """ comment -> I can create the newDataset in any workspace I want. """ fct.create_blank_semantic_model(newDatasetName, workspaceName='DevWS')

time.sleep(5)

""" comment -> re-creates all tables and columns of source_semantic_model (=param 1) in target_semantic_model (=param 2) -> lakehouseName and lakehouseWorkspaceName define the workspace and lakehouse where the data should be coming from. (Switching datasource, all fine!)

-> PROBLEM: It does not seem to work if source- and target-semantic-model do not reside in the same workspace! """ fct.migrate_tables_columns_to_semantic_model( 'sourceSM', 'targeSM', workspaceName= 'DevWS', lakehouseName= 'LH_Gold', lakehouseWorkspaceName='ProdWS')

fct.migrate_model_objects_to_semantic_model('sourceSM', 'targetSM', workspaceName= 'DevWS')

I am not sure if that's an issue. Could be a feature.

Thx a million for a feedback.

Best Regards from Germany, Marc

— Reply to this email directly, view it on GitHubhttps://github.com/m-kovalsky/fabric_cat_tools/issues/2, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHBQBNXTF6QZV757VJ3TYP3Y4U2KBAVCNFSM6AAAAABGANGHDOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZTKNJVGM2TKMY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

MarcAHunt commented 7 months ago

Great. Thx. Any timeline?

Best Regards, Marc

m-kovalsky commented 7 months ago

Would you try using a beta version of 0.3.0 in this folder, along with the accompanying notebook. Note that parameter names have changed so following the new notebook will help with that.

https://github.com/m-kovalsky/fabric_cat_tools/tree/main/testing

MarcAHunt commented 7 months ago

It works like charm. Can't emphasis how much this is appreciated.

My test-case is as follows: In a dev-workspace I build a lakehouse and from that a new semantic-model and report.

Now, I want to bring it in production (separation of data and report workspaces) I am setting these vars:

dataset_name = 'my_dev_semantic_model' #Original or source semantic-modelworkspace_name = 'my_dev_workspace' #Workspace were source semantic-model residesnew_dataset_name = 'my_new_semantic_model_to_be_created' #New target semantic-model to be creatednew_dataset_workspace_name = 'my_prod_workspace' #Workspace were new target semantic-model is to be created

*lakehouse_name = 'my_prod_lakehouse' #Production lakehouse, of course holding the same schema as dev-lakehouselakehouse_workspace_name = 'my_prod_workspace_for_lakehouse' #Workspace of prod-lakehousenew_report_name = 'my_new_report_copied_from_dev_report_but_still_bound_to_dev_semantic_model'

New report to be bound to new semantic-modelnew_report_workspace_name =

'my_prod_report_workspace' #Workspace of report*

My findings:

  1. In your example notebook rebinding one report is documented like this:

fct.report_rebind( report = report_name ,new_dataset = new_dataset_name ,workspace=workspace_name)

I found these working parameters to run it against a different workspace: f

ct.report_rebind( report = new_report_name ,dataset = new_dataset_name ,dataset_workspace = new_dataset_workspace_name ,report_workspace=new_report_workspace_name) Would be great if example could be adjusted to all possible (and optional) parameters

  1. The URL to install the beta-version is missing the testing directory. Instead of this: %pip install " https://raw.githubusercontent.com/m-kovalsky/fabric_cat_tools/main/fabric_cat_tools-0.3.0-py3-none-any.whl " it needs to be this: %pip install " https://raw.githubusercontent.com/m-kovalsky/fabric_cat_tools/main/testing/fabric_cat_tools-0.3.0-py3-none-any.whl "

But hey! You saved my day and a lot of manual work in my job. Thank you so much, Marc

Am Mo., 15. Apr. 2024 um 21:40 Uhr schrieb m-kovalsky < @.***>:

Would you try using a beta version of 0.3.0 in this folder, along with the accompanying notebook. Note that parameter names have changed so following the new notebook will help with that.

https://github.com/m-kovalsky/fabric_cat_tools/tree/main/testing

— Reply to this email directly, view it on GitHub https://github.com/m-kovalsky/fabric_cat_tools/issues/2#issuecomment-2057669442, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHXG2MMLG7YABE67XGOCZNDY5QUJJAVCNFSM6AAAAABGANGHDOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJXGY3DSNBUGI . You are receiving this because you authored the thread.Message ID: @.***>

-- Marc A. Hunt Rainerstr. 7 83059 Kolbermoor Mobil: +49 (0) 172 176 06 59 Email: @. @.>mail.com

m-kovalsky commented 7 months ago

Great to hear and thank you for testing! The notebook and documented parameters will all be updated once I release this new version. The pip install was still aligned to the main site instead of /testing so glad you used the correct pip install link.

MarcAHunt commented 7 months ago

You are very welcome.

One more question: Do you see any possibility to make this work for semantic-models based on warehouses instead of lakehouses? Currently it does seem to work.

Thx

Am Di., 16. Apr. 2024 um 09:59 Uhr schrieb m-kovalsky < @.***>:

Great to hear and thank you for testing! The notebook and documented parameters will all be updated once I release this new version. The pip install was still aligned to the main site instead of /testing so glad you used the correct pip install link.

— Reply to this email directly, view it on GitHub https://github.com/m-kovalsky/fabric_cat_tools/issues/2#issuecomment-2058472765, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHXG2MMH254KFFAS3UFH3FLY5TK4PAVCNFSM6AAAAABGANGHDOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJYGQ3TENZWGU . You are receiving this because you authored the thread.Message ID: @.***>

-- Marc A. Hunt Rainerstr. 7 83059 Kolbermoor Mobil: +49 (0) 172 176 06 59 Email: @. @.>mail.com

m-kovalsky commented 7 months ago

This is intended for migrating to Direct Lake which is predicated on a lakehouse, not a warehouse. If you are thinking of another idea, please explain further.

Btw, 0.3.0 is available and the function documentation has been updated.

From: MarcAHunt @.> Sent: Tuesday, April 16, 2024 10:14:08 AM To: m-kovalsky/fabric_cat_tools @.> Cc: Michael Kovalsky @.>; Comment @.> Subject: Re: [m-kovalsky/fabric_cat_tools] migrate_tables_columns_to_semantic_model - source- and target semantic models in different workspaces (Issue #2)

You are very welcome.

One more question: Do you see any possibility to make this work for semantic-models based on warehouses instead of lakehouses? Currently it does seem to work.

Thx

Am Di., 16. Apr. 2024 um 09:59 Uhr schrieb m-kovalsky < @.***>:

Great to hear and thank you for testing! The notebook and documented parameters will all be updated once I release this new version. The pip install was still aligned to the main site instead of /testing so glad you used the correct pip install link.

— Reply to this email directly, view it on GitHub https://github.com/m-kovalsky/fabric_cat_tools/issues/2#issuecomment-2058472765, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHXG2MMH254KFFAS3UFH3FLY5TK4PAVCNFSM6AAAAABGANGHDOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJYGQ3TENZWGU . You are receiving this because you authored the thread.Message ID: @.***>

-- Marc A. Hunt Rainerstr. 7 83059 Kolbermoor Mobil: +49 (0) 172 176 06 59 Email: @. @.>mail.com

— Reply to this email directly, view it on GitHubhttps://github.com/m-kovalsky/fabric_cat_tools/issues/2#issuecomment-2058502887, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHBQBNRX3KZHV4P4MOEVFO3Y5TMVBAVCNFSM6AAAAABGANGHDOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJYGUYDEOBYG4. You are receiving this because you commented.Message ID: @.***>