search

RuiRomano / pbimonitor

MIT License
207 stars 76 forks source link

Error 429 - Throtteling #48

Closed AdminpowerBI closed 3 months ago

AdminpowerBI commented 4 months ago

Hello, I work for a company with 120K workspaces, including personal spaces. I can only recover 50K workspaces and then I get error 429 - Throtteling.

I switched to Plan premium on azure functions but it didn't help.

Do you have a patch to get the WS from the whole tenant power bi?

Thank you very much.

WillCisler commented 4 months ago

The scanner API has a limit of 50k workspaces an hour so the solution will need to pause for an hour to clear the API limit.

The code is built to do this automatically and it can run on very large tenants. It does from time to time throw new error messages that are rooted in throttling but come up as errors further up the stack (JSON, server 500s).

Can you confirm:

Have you changed the timeout limit in the deployed as an azure function? For a tenant of 120 WS you will need at least 3 hours.

What error are you getting?

AdminpowerBI commented 4 months ago

Hello, I raised the timeout for the app to 3h30. I laucnhed another full scan, but after 51K WS, i got this error. Did i do something wrong ?

I think that the execution of the script doesn't restarted automatically.

Thank you in advance for your help

Here are all the conditions info for the test: AppPlan Config : app_plan

Function Timeout config : Function_timeout

Throtteling Config : config_throttle handling variables

Error + Script doesn't restarted after 78minutes : error_throtteling_catalog script doesn't restarted

WillCisler commented 4 months ago

Can you compare the code you have to the main branch of the Repo? The last merge handled that error specifically.

https://github.com/RuiRomano/pbimonitor/commit/f134caaa718ef8529c1ba624a4da6969c08ee778#diff-d0030e63e613f48f594af7f352dd3e0990d1ae347d242525fee75c79ffbd1372

JamesDBartlett3 commented 3 months ago

@AdminpowerBI, are you still experiencing this issue, and if so, have you already updated to the latest version of pbimonitor as @WillCisler suggested above?

AdminpowerBI commented 3 months ago

@WillCisler & @JamesDBartlett3, I did update the scripts but it didn't fix the problem.

However, I've found a workaround that consists in adding an 8-second sleep to the "GetArrayInBatches" function. This spread the execution of the script over 3 hours and thus bypassed error 429 throtteling.

I succeeded in recovering all 104K Workspaces.

Thank you for your support and I apologize for the delay in replying.

image

Sincerely