search

microsoft / ProjectAcoustics

Microsoft Project Acoustics
https://aka.ms/acoustics
Creative Commons Attribution 4.0 International
135 stars 21 forks source link

Bake stuck on "Initializing Compute Nodes..." UE4 #78

Closed RAI-Clara closed 2 years ago

RAI-Clara commented 2 years ago

When trying to bake my audio scene, it seems to get stuck on "Initializing Compute Nodes...".

I've looked at problem #72, and requested 48 cores for Standard FSv2 vCPUs Virtual Machine for my azure batch, but it still get stuck at that point. I checked the pool for my batch account, and it still says resize error, account Core Quota reached. Did I do something wrong in setting up the Quota increase, or is it my settings in UE?

Right now my settings for UE bake are F8sV2 virtual machine, and 6 Nodes

MikeChemi commented 2 years ago

Sounds like there's nothing wrong in your UE settings. Can you confirm that your quota increase has been approved and the new quota is reflected in your account?

RAI-Clara commented 2 years ago

This is the first time I've tried requesting an increased quota so it could be that.

When I go into my Azure subscription, under usage + quotas, I see for Standard Family vCPUs I have 0 of 48 in the far right column. I changed the limit to 48, so I was assuming that would increase the quota?

MikeChemi commented 2 years ago

Maybe? More questions. Did you restart your job after requesting more quota, or is it the same job? I think you need to delete the old job and start a new one for the new quota to go in to effect.

RAI-Clara commented 2 years ago

hmm I believe I deleted the old job. I clicked on cancel inside Unreal engine, and whenever I do this in my batch it says there are no jobs currently.

MikeChemi commented 2 years ago

OK, sounds correct.

I'm refamiliarizing myself with the documentation to request quota increases. It says support will contact you, and it may take up to 2 business days. Have you been in touch with support yet? It sounds like the quota increase is not yet processed.

RAI-Clara commented 2 years ago

Ah no not yet, I requested the quota increase yesterday, so they could still be processing it then. I guess I'll give it some more time and see if they get back to me about the increase.

Thank you, I'll keep you updated!

RAI-Clara commented 2 years ago

Unfortunately It's still happening, I keep getting the error "There was an error encountered while performing the last resize on the pool. Please try resizing the pool again." And I still haven't gotten an email about the quota increase yet either. Even if I try to request 48 cores again, it says I already have them.

MikeChemi commented 2 years ago

Sorry to hear there's been no progress :( When you get stuck in this state, can you navigate to the compute pool in the Azure portal and see if there's any resize errors displayed?

MikeChemi commented 2 years ago

Oh, sorry, I re-read your last comment and realized I misinterpreted it. I'll do some digging and reply back shortly.

MikeChemi commented 2 years ago

Is that resize error clickable? If you can click on it, does it give any more detailed information, or are you stuck with "please try again"?

RAI-Clara commented 2 years ago

You're fine haha When I click on it I get:

Resize errors - (Pool has 1 resize error)

There was an error encountered while performing the last resize on the pool. Please try resizing the pool again. Code: AccountCoreQuotaReached

Message: The specified account has reached core quota

Values: n/a

MikeChemi commented 2 years ago

This is great information. You're still having quota issues. At this point I'd recommend opening a (new) support ticket with this error and see if you can get someone from support to fix your quota.

RAI-Clara commented 2 years ago

Alright I'll try that, thank you. If I find a solution, I'll be sure to post an update.

RAI-Clara commented 2 years ago

I wasn't able to find a solution to this, so my azure account was upgraded to Dev so I could use support tickets. And then when I requested a quota increase again, it all worked.

I tried baking my scene, and got the ace file, but unfortunately it doesn't seem to be affecting the audio. When I go behind a wall it sounds the same as if it wasn't there. I added all the static meshes and the nav mesh, and I'm pretty sure I followed all the steps correctly. Any advice?

MikeChemi commented 2 years ago

Great! Glad the quota issue got sorted out.

First thing to do when debugging why it doesn't sound how you expect is to turn on all the debug information. Make sure the probes are being loaded in, and there are voxels where you expect, by turning on the debug drawing on the acoustics space actor. After that, turn on the debug drawing for acoustics audio components, too, and see if there are any error messages displayed.

RAI-Clara commented 2 years ago

yep, one step closer! :D

I turned on the debug draw and as far as I can see there are no errors popping up from that. But I do see an error "LogAkAudio: Error: Plug-in unsupported channel configuration: 69942" in the output log of Unreal

MikeChemi commented 2 years ago

Ah! That's a good clue. What settings do you have set in the mixer plugin inside Wwise, and what is your audio device setting in Windows?

RAI-Clara commented 2 years ago

hmmm for Wwise I have:

And for Windows Device Settings:

RAI-Clara commented 2 years ago

OK got the error gone, I think it had to do with the 7.1 Config. When I changed it back to same as parent it all works, so I'm guessing there was an issue having mismatching channels. Maybe that can be fixed with a channel router in case anyone wants mismatching channels? (just guessing from what I've read)

So that's sorted, but now there is the error voice starvation, but I only have 2 audio events in a scene. It's a little random when it does it; after I changed the attenuation settings in the wwise sound settings to a lower max distance the error happens less, but still occasionally pops up.

Also I still can't hear any PA effects even after fixing the error :(

MikeChemi commented 2 years ago

The HRTF Spatialization mode only supports stereo output. If you want to support 7.1, you'll need to change to panning.

Error Voice Starvation is actually expected when the plugin starts up. (Weird, I know) so that's a good sign! If you're not seeing it at all, that means the plugin never clicked on.

Just double checking - the output of your actor mixers is set to your PA bus, correct?

RAI-Clara commented 2 years ago

Yes, I believe so here's a quick snapshot of one of them. img

MikeChemi commented 2 years ago

OK, looks good. When you say "can't hear any PA effects" - which effects are you expecting? i.e. do you have walls you're walking behind, but don't hear occlusion? Have you configured your occlusion curves in Wwise? Does the debug drawing in UE show that values are changing for your sound source?

RAI-Clara commented 2 years ago

Huh ok, it's actually working now! After fixing the one issue I restarted UE4 and tried rebaking the scene and it's working. So as far as I know with this quick lookover, I think it's all working, thank you so much!

Also, the issues I was having was when the audio device was behind a wall or in the large room I could hear both of them as if there was no wall there. I also couldn't hear any reverberations in the room. I'll attach some snapshots of the setup. audio1 Audio2

Do I need the occlusion curves? We're trying to use PA as a true to life (or as close as possible) audio simulation, so would I need to change anything to do that? Also does PA do delayed sounds; as in over a long distance would the sound be offset at all?

MikeChemi commented 2 years ago

Hooray! Glad things are working :)

You'll want to set the occlusion curves as they are in the documentation. This will give you the most realistic sound.

PA does not delay sounds based on distance.

MikeChemi commented 2 years ago

Closing this issue since baking is working. Feel free to open new issues for any other questions that pop up.

RAI-Clara commented 2 years ago

I'm having the same issue again, but the job actually submits and is computing, but the issue is it never stops. I assumed it would stop after the estimated 1 hour of bake time, but it kept running for the past day; I was also charged for the service it seems. And sadly I never even got the baked scene out of it, it was still stuck on initializing nodes.

This happened after I tried integrating PA into a previous project we had, and I have everything set up the same way as before.
Could it be a python issue? I recently modified some python modules for an assignment in one of my classes.

Also, since the bake was never successful, is there anyway to get a refund for the costs of the bake? It was running far past the estimated bake time and accumulated a fair amount of charges. :(

MikeChemi commented 2 years ago

Sorry to hear you're still having trouble. You said some things that appear to be conflicting though. Can you help get me sorted out?

"the job actually submits and is computing"

What did you mean by this? It sounds like the bake was progressing, not stuck on initializing.

"I never even got the baked scene out of it, it was still stuck on initializing nodes."

This sounds like the original issue - compute pool stuck on initializing nodes. However, if the nodes aren't allocated yet like they were previously, you won't be charged.

When you go to your Batch account in Azure portal, what is the state of the pool? Are the nodes showing they are OK and computing? Are there any errors?

"is there anyway to get a refund for the costs of the bake?"

Depends on what the issue is. Let's get to the bottom of what happened. Any refunds would be issued as a result of opening a support ticket, and the more information we can give them the better.

RAI-Clara commented 2 years ago

When I ran bake, "UE4 says initializing compute pool nodes...", and when I go to azure portal under jobs, I can see there is a job running, and inside there are 160 tasks currently active. I saw this, and left the thing running, and then I found out yesterday that the computation was still going; there was no ace file created either.

I'm unsure if this all is the same status as before on the Feb 28th - March 1st though, this is just what it's doing now when I run it. Also from what I can see I was charged $35.30 for March first, and $15.89 for the Feb 28th.

MikeChemi commented 2 years ago

OK, thanks for the extra information. Something fishy is going on - you should be able to see pools and nodes under the Pools blade. I've asked @ashtat to help you out with this issue. In the meantime, you can try opening a support ticket to see if the Batch support crew can help figure out what's going on with your compute nodes.

RAI-Clara commented 2 years ago

Ah when I go to the Pools tab in the batch account I can see 6 dedicated nodes, 48 current vCPUs and allocation state steady. When I click on the blue boxes in the node heat map section I see they are in the state "starting," and they actually just all started turning pink and now they say state: "unusable."

Sounds good, I'll set up a support ticket in the meantime. Thank you!

ashtat commented 2 years ago

@RAI-Clara sorry to hear you're having these issues. I'm curious if you ever saw any nodes transition to started? This could be a Batch infrastructure issue for which we'll need to engage with Azure Batch support team. But in case you notice any nodes actually do transition to "started" then we can pull some logs and check if something is going awry in the bake tools.

One quick thing to check though, we also noticed this on our end with one of our jobs that the nodes were failing to start due to incorrect containr image URL case. To check for this,

Do you see the node properties reporting any errors? For example, in our case, we see this error -

image

RAI-Clara commented 2 years ago

Yes when going to the node's properties I'm getting the same error.

ashtat commented 2 years ago

Ah, can you go into the bake tab in the editor, make sure that the url in "Project Acoustics Toolset Version" is all lowercase and then resubmit? There was a change in the container registry infrastructure that made this url case-sensitive.

RAI-Clara commented 2 years ago

Yep that worked, thank you! Could this issue have also caused the processing to never end, and created the charges I'm seeing on the account? I can see the charges are from the virtual machines processing from the February 28th - March 1st.

ashtat commented 2 years ago

Unfortunately, I think Azure will still charge you for unstable VMs. Have you filed a ticket for this? We can reach out to the Batch support team from our end to see if they can issue a refund.