search

RuiRomano / pbimonitor

MIT License
221 stars 83 forks source link

High memory usage when using local PowerShell solution #22

Closed aditya-anand-chegu closed 2 years ago

aditya-anand-chegu commented 2 years ago

Hi @RuiRomano, I had implemented the local PowerShell solution for a large tenant in April. The solution was working fine till about August end, after which it was running into time-out issues. Trying to run the Power BI template on Power BI Desktop throws an error saying there is no available memory, even for machines as large as 678 GB RAM. Is there any way to optimize the data load to improve refresh performance, since incremental refresh is not possible on the local version of the solution?

RuiRomano commented 2 years ago

Thanks for the feedback. Just to be clear the large memory consumption is at the dataset/pbix refresh and not powershell scripts, correct? I made some updates in August that should optimize because it was reading the scanner files multiple times. But unfortunately as you said its not possible to implement incremental refresh in Desktop the only solution is to skip/filter some files (ex: only look at X days/months of data using the Range Start/End parameters). Other alternative is to use a Dataflow, I also deployed a dataflow version recently.