Pipeline "Load_Inventory_E2E" is failing

FrankPreusker commented 4 weeks ago

During execution of the notebook "01_Transfer_Incremental_Inventory_Unit" is stops in cell 10 "#Get API data" with the error:

FabricHTTPException: 401 Unauthorized for url: https://api.fabric.microsoft.com//v1.0/myorg/admin/workspaces/modified?excludePersonalWorkspaces=True&excludeInActiveWorkspaces=True Headers: {'Cache-Control': 'no-store, must-revalidate, no-cache', 'Pragma': 'no-cache', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/octet-stream', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'RequestId': '52ebc815-8bc9-4b7c-b383-b9d28ec56069', 'Access-Control-Expose-Headers': 'RequestId', 'Date': 'Thu, 24 Oct 2024 21:42:04 GMT'}

We're already extracting Scanner API data with the same Service Principle provided for this solution. The only difference is, that we do this via a PS script (the solution from Rui Romano) and I believe this is utilizing the API "https://api.powerbi.com/v1.0/myorg/admin/workspaces/modified" instead.

kethom-analytics commented 3 weeks ago

Can you please check, if this Service Principal has any API permissions in Entra? We saw some errors in the past, when there were API permissions configured. Additionally can you verify, that the SP authorized for both Admin settings:

Service Principals can use Fabric APIs
Service Principals can access read-only admin APIs

If you can tell me your exact setup, i will test it in my tenant.

Best regards Kevin

FrankPreusker commented 3 weeks ago

As we're using that SP also to extract user & groups information, it does have two MS Graph permissions: Directory.Read.All and User.Read.All
The settings Service Principals can use Fabric APIs & Service Principals can access read-only admin APIs are activated and the SP is part of an allowed Security Group

PS: The List Items API is actually working with the same connection (fabric-service-api admin).

daanriepma commented 3 weeks ago

It appears that no authentication is performed in this step, and the created connections are not actually being used. I am encountering the same issue. Since I am not an admin in the Azure environment, I am also receiving a 401 error.

kethom-analytics commented 3 weeks ago

@daanriepma You are right. We are using sempy to do the request within the notebooks. There it uses the executing users identity. Since we are admins on our environment, we missed that. Let me think of a better alternative

daanriepma commented 3 weeks ago

@kethom-analytics, I will also build a solution on our end (REST API authentication with Key Vault integration) and will share it with you afterward.

kethom-analytics commented 3 weeks ago

@daanriepma I was also thinking about this. Using mssparkutils to get the Key Vault secrets and then getting a token for the SP. But in this case we would still use the executing users identity to query the key vault. I guess this is still better than requiring the user to have admin rights, but i was thinking, if there might be another way

FrankPreusker commented 3 weeks ago

Here is an implementation example from klinejordan

kethom-analytics commented 2 weeks ago

Hello,

I included the service principal authentification into the notebook and also adapted the 429 handling from the script you from klinejordan. It will be included in the next version

GT-Analytics / fuam-basic

Pipeline "Load_Inventory_E2E" is failing #5