RuiRomano / pbimonitor

MIT License
221 stars 83 forks source link

429 issue for data catalog in large tenant #33

Closed FrankPreusker closed 1 year ago

FrankPreusker commented 1 year ago

When executing a full run of the scanner API (catalog.ps1) without the personal workspaces the script is running perfectly fine. However, adding the personal workspaces gets us to ~70k workspaces and therefore over the throttling limit of 500 (x 100 workspaces) requests per hour. I'm not any expert with PowerShell, but I was assuming that the script is taking care of that with the line 239 in script catalog.ps1:

Wait-On429Error -tentatives 1 -sleepSeconds $throttleErrorSleepSeconds -script {

However, the script is stopping with the known 429 error message:

C:\PBIMonitor\Fetch - Run.ps1 : Error on 'C:\PBIMonitor\Fetch - Catalog.ps1' - System.AggregateException: Mindestens ein Fehler ist aufgetreten. ---> System.Net.Http.HttpRequestException: Der Antwortstatuscode gibt keinen Erfolg an: 429 (). bei System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() bei Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.d35.MoveNext() --- Ende der internen Ausnahmestapelüberwachung --- bei System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) bei Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet() bei Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord() ---> (Interne Ausnahme #0) System.Net.Http.HttpRequestException: Der Antwortstatuscode gibt keinen Erfolg an: 429 (). bei System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() bei Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.d35.MoveNext()<---

1) Should the script take care of the 500 request/hour limit? Why is it not working correctly? 2) If not, how can we get around it?

FrankPreusker commented 1 year ago

This was due to the German PowerShell script version. Once we moved to the English version its running perfectly fine.