search

Insight-Services-APAC / APAC-Capability-DAI-DbtFabricSparkNb

MIT License
8 stars 3 forks source link

Capability to generate notebooks that can be checked into Fabric version control #24

Closed jp-vanheerden closed 2 months ago

jp-vanheerden commented 4 months ago

Framework generates .ipynb files that can be imported into Fabric, but when these files are version controlled the are saved as .py files in a folder:

image

Ideally I'd like the process to be:

  1. generate notebook
  2. copy notebook to local version controlled folder
  3. check notebook into version control
  4. refresh Fabric workspace and notebook is available

If this isn't possible, preference would be to (manually) import notebook into Fabric rather than import via lakehouse

insightgrantkrieger commented 3 months ago

I tried to test but could not get access into DEVOps for Demo org for grant.krieger@insight.com. Tulp was away

insightgrantkrieger commented 3 months ago

A new method has been added in utils.py called IPYNBtoFabricPYFileAndGitStructure where the bulk of the coding has been done.

The process looks at the output files in /target/notebooks and converts to /target/notebooks_fabric_py. This structure is ready to be uploaded into git

insightgrantkrieger commented 3 months ago

In Fabric update new notebooks. There is a work around where you need to disconnect git in Fabric workspace and re-connect to new directory to sync new notebooks otherwise you get a "Git integration error 'No files to commit'". This has been logged https://community.fabric.microsoft.com/t5/Issues/Git-integration-error-No-files-to-commit-after-add-new-notebook/idi-p/4035845#M108345

annstephyjose commented 3 months ago

I've tested the change #24 in the feature branch, which resulted in Python files being generated under the 'notebooks_fabric_py_git' folder. However, I couldn't verify the synchronization with devops-fabric-git due to permissions limitations using my fabric profile. I discussed this with Grant and confirmed the results on his system.

insightgrantkrieger commented 3 months ago

Note: Updated work around steps: Disconnect repo Delete all newly added files in repo only folder (Do not delete all files only delete new files) Then reconnect to repo same repo folder and all good

insightgrantkrieger commented 3 months ago

Another new method has been added in utils.py called IPYNBtoFabricPYFile where the bulk of the coding has been done.

The process looks at the output files in /target/notebooks and converts to /target/notebooks_fabric_py. This structure is ready to be uploaded into a fabric API.

insightgrantkrieger commented 3 months ago

APIUpsertNotebooks has also been added which easily updates notebooks directly to the API end point

insightgrantkrieger commented 3 months ago

Aligned to Dev branch and John's latest changes for pip install

insightgrantkrieger commented 2 months ago

This is covered by api update. File structure has been utilized for that