search

microsoft / ProjectAcoustics

Microsoft Project Acoustics
https://aka.ms/acoustics
Creative Commons Attribution 4.0 International
138 stars 21 forks source link

Difference between estimated and actual bake times (cloud bake) #146

Open PDavidFL opened 1 year ago

PDavidFL commented 1 year ago

Hi team,

First of all, thank you for your hard work on this super cool project! My team and I think it's a real game changer. :)

I'd like ask about a finding that we have since we started to experiment with cloud baking for larger areas. What we found is that the estimated bake times are quite far from the actual time spent on the processing.

For example, for our last bake, we requested 25 (F8s V2) nodes for about 7700 probes. The 'Estimated Bake Time' showed 27 min, and the 'Estimated Compute Cost' was 11 h 30 min. Unfortunately the bake took about 6.5 h and costed about 13 times more as initially calculated... Similar differences happened to our previous two bakes as well. (We use UE5 with Wwise 2021 and have the latest PA version installed.)

According to Azure Monitoring, the number of utilized nodes was more or less staying on 25, so that's probably not the problem.

Of course, these are just estimates but we are wondering if the differences we experience (more than 10x) are normal or there is some problem with our configuration or something else?

A guess from my side is that the actual bake ignores our custom Simulation Region setup for some reason? (Ours: [4000 x 4000 x 1400] vs Default: [9000 x 9000 x 3500] --> ~12.7x diff )

Cheers, David

NoelCross commented 1 year ago

@PDavidFL We do take into account the simulation region when estimating the costs. If you would be willing to send us your _config.xml file, we'd be happy to take a look at why the results you are seeing are off by so much. You are correct that we don't test the estimation of non-standard settings all that much so you might have uncovered something that we need to fix. Sorry for the misleading estimations results you are seeing.

PDavidFL commented 1 year ago

Hi Noel, sure, thanks for checking this out! LVL_NV04_Partitions_02_AcousticsData_config_xml.txt

PDavidFL commented 1 year ago

Another discovery that might help investigating this: we have performed a new modified bake from the same game level, and the bake time was very close to the estimated value this time!

The changes we made on the config:

Attached the config file for this one as well: LVL_NV04_Partitions_03_AcousticsData_config_xml.txt

NoelCross commented 1 year ago

@PDavidFL Thank you for providing the additional configuration information. I haven't yet had time to look at this, but I have logged this issue in our internal tracking database and will look into this again when the time is right in our development cycle. Thank you for reporting this.