Pipeline "Load_Items_E2E" is failing

[FrankPreusker]

The execution of the pipeline "Load_Items_E2E" is failing with 429 error message:

[kethom-analytics]

Hello,

Do you roughly know how many items you have within Fabric? The API supports up to 200 requests per hour extracting 10000 items per request, which should allow a extraction of up to 2 million items (https://learn.microsoft.com/en-us/rest/api/fabric/admin/items/list-items?tabs=HTTP)

Have you tried it only once or multiple times?

Thanks in advance Kevin

[FrankPreusker]

We are already using the Scanner Data API to extract the metadata. That way we get ~130k artifacts - despite a lot this is way below the above mentioned API limit.

Next I have limited the API in the Pipeline step "Get Items" to filter for specific types only, eg items?type=SemanticModel This did work as it should and retrieved 57200 records. I've also used some of the other item types from the api documentation with varying response; eg type=PaginatedReport is failing with an error code 400 BadRequest 🤔, type=Report was ok returning another 55000 records.

General question: why do you consider the Get Items REST API in addition to the Scanner API as I don't see that it will deliver anything else? (maybe I'm missing something here)

PS: during testing I've noticed a bug for the history archiving, but will raise this as a new issue.

[ggintli]

Hello @FrankPreusker,

The idea with the Get Items API is that - it returns automatically every (supported) item types and it is a great candidate to have easily a generalized table for all the items, which makes easier the data modeling later on.

I see your point with that Scanner API too. Of course it is your decision to choose, which is better in your case, we tried to provide more possibilities and options to our FUAM Basic users.

Still strange though that you receive this issue. We will investigate on that if it is on FUAM side.

Best regards, ggintli

[kethom-analytics]

Hello @FrankPreusker,

It seems this error is caused by an additional throtteling for the Fabric API which is independed of the documentation i pasted above: Fabric API Throtteling

When you increase the "Request interval (ms)" to 10.000, the error doesn't seem to occur. I tested it with a customer who had the same issue. Here this helped to resolve it. I will increase this value in the next version and will check if there is another way to prevent the error as well

[FrankPreusker]

I can confirm that increasing the "Request interval (ms)" to 10.000 did resolve the issue for me as well :-)