Neos Randomly Hanging and getting World Crashes

AshtonSparx commented 2 years ago

Describe the bug?

Neos BLFC world, bunch of user's neos starts hanging but not for everyone. On my headless it's happened as well. I'll be supplying everyone's log that was present. We're not 100% certain what exactly is causing it. There doesn't seem to be a common occurance. I did notice twice when I went into free cam mode, everyone instantly got a world crash and had to rejoin.

Relevant issues

Possibly #3244 ??

To Reproduce

Like I mentioned before, I have no idea how to reproduce hence supplying a bunch of user's logs.

Expected behavior

For Neos to not hang without a crash error.

Log Files

Mine headless's Logs: DESKTOP-SPEUN1P - 2021.10.26.9 - 2021-10-25 21_14_10.log

My User Log:

[DESKTOP-SPEUN1P - 2021.10.26.9 - 2021-10-28 00_2700.log](https://github.com/Neos-Metaverse/NeosPublic/files/7431892/DESKTOP-SPEUN1P.-.2021.10.26.9.-.2021-10-28.00

DESKTOP-SPEUN1P - 2021.10.26.9 - 2021-10-28 02_47_08.log 27_00.log)

My Player Logs: Player.log Player-prev.log

My logs when my game hung in BLFC pool world: Player.log DESKTOP-SPEUN1P_-2021.10.26.9-_2021-10-27_21_12_08.log

Soap's Log:

DESKTOP-M3LTEEK_-2021.10.26.9-_2021-10-27_21_03_13.log

Screenshots

No response

How often does it happen?

Sometimes

Does the bug persist after restarting Neos?

Yes

Neos Version Number

2021.10.26.9

What Platforms does this occur on?

Windows

Link to Reproduction Item/World

No response

Did this work before?

No

If it worked before, on which build?

No response

Additional context

If anyone else has log files of when Neos hangs and doesn't return or you suddenly get a world crash error. Please contribute some logs!

Your player log can be found: \AppData\LocalLow\Solirax\NeosVR Your normal log can be found: \steamapps\common\NeosVR\Logs

Reporters

AshtonSparx#5050 AdmiralSoap#4934

Frooxius commented 2 years ago

I see one issue in one of the logs so far, but not much else. I'll try to fix that one up, that should help some.

Some of the logs you posted failed to upload

How do the crashes manifest exactly? Is it always a world crash? Or something else?
Did your headless crash too?
Does this happen specifically in the BLFC world and not any other?

AshtonSparx commented 2 years ago

I see one issue in one of the logs so far, but not much else. I'll try to fix that one up, that should help some.

Some of the logs you posted failed to upload
* How do the crashes manifest exactly? Is it always a world crash? Or something else?

* Did your headless crash too?

* Does this happen specifically in the BLFC world and not any other?

Which logs failed to upload? I can try again or upload to a hidden GH file thingy.

For the other questions

How do the crashes manifest exactly? Is it always a world crash? Or something else?

The world crashes instantly kicks the user from the world and sends them to their last focused world. The game doesn't actually full crash. It'll just forever stay not responding. If you're in VR. It just stays on "Waiting on Neos"

My headless doesn't seem effected by it. It just stays open as normal.

This happened in other worlds, not just headless's.

Frooxius commented 2 years ago

@AshtonSparx The links for your own user logs seem to be a bit garbled.

What's the timeline on the crash? If I understand it, they get kicked into last focused world and then Neos freezes? Is there a delay between getting kicked out and freezing?

AshtonSparx commented 2 years ago

From what I've witnessed personally, the getting kicked and freezing are separate from each other.

The freezing seems to be like Neos having a blindfold and shotgunning into a crowd. In the BLFC world. A handful of users would all at the same time freeze out of no where. Seems to possibly be tied to users joining/leaving. From what Epsilion was gathering. It wouldn't always be the same users, but other users also wouldn't freeze even if others did several times during the same session.

AlexThorin commented 2 years ago

Contributing my logs to this. Interestingly, mine might have been something else, since it was locking up and slowing down the rest of my computer too even though it was at 50% cpu and had about 6 GB of ram free. Once neos was closed everything went back to normal. Not sure if neos just started DUMPING stuff into logs at a super high rate? Dunno how to read logs though so i'll just let you guys figure it out

Player.log FOXSTATION - 2021.10.26.9 - 2021-10-28 02_46_06.log

Feels like the same outcome though. Can still talk and are still connected, but everything's frozen

partygrunt commented 2 years ago

Here are my logs, however I can not find any concrete relationship between when I would crash and what was happening in the world. The world would just hang in the current session on the "waiting" screen but you could freely talk with the individuals until you decided to close out of the session.

Player-prev.log Player.log

Frooxius commented 2 years ago

I've added an exception guard and some extra diagnostics in 2021.10.28.1280. Can you please test with that build and see if there are any improvements?

Please share any new logs if you run into issues (even if it doesn't fully crash).

AshtonSparx commented 2 years ago

I've added an exception guard and some extra diagnostics in 2021.10.28.1280. Can you please test with that build and see if there are any improvements?

Please share any new logs if you run into issues (even if it doesn't fully crash).

Will do this immediately! We'll crunch this mystery together! Thanks Mr. Froox! Will keep you updated.

bontebok commented 2 years ago

I too can report I am having this issue. It happens on average about once every 2 hours. Another user said it may be my Vive Face Tracker as that hardware is not typical for many users. I can report that after about 3 days without the face tracker plugged in, I am still experiencing Steam Waiting... hangs.

One thing to note, the users can still hear me in the world and I can hear them however no display updates occur. Having the host kick me from the world still leaves me "in" the world, meaning I still hear ambient audio playing but users can't hear me and I cannot hear them.

If there's anything I can add to be helpful to diagnosing this, let me know. Rucio#0134 on Discord.

AshtonSparx commented 2 years ago

I too can report I am having this issue. It happens on average about once every 2 hours. Another user said it may be my Vive Face Tracker as that hardware is not typical for many users. I can report that after about 3 days without the face tracker plugged in, I am still experiencing Steam Waiting... hangs.

One thing to note, the users can still hear me in the world and I can hear them however no display updates occur. Having the host kick me from the world still leaves me "in" the world, meaning I still hear ambient audio playing but users can't hear me and I cannot hear them.

If there's anything I can add to be helpful to diagnosing this, let me know. Rucio#134 on Discord.

Is this still happening on Neos Version 2021.10.28.1280?

bontebok commented 2 years ago

Is this still happening on Neos Version 2021.10.28.1280?

It was, but I think I may have resolved it as I was able to be in world for at least an hour and a half without a crash, however I need to do more testing to see if I am completely void of the problem.

The steps I took were to completely uninstall SteamVR and Neos, insuring that I deleted the AppData\LocalLow\Solirax\NeosVR folder which included the Neos.lightdb and Assets folder. Once I reinstalled SteamVR and Neos, I had to login and run through the tutorial but after that things seemed to have improved.

I'll report back tomorrow my results.

AshtonSparx commented 2 years ago

Confirmation, it still exists. The moment someone joined the world, everyone's Neos hung except for the host. DESKTOP-SPEUN1P - 2021.10.28.1280 - 2021-10-28 17_46_55.log

Player.log

This is a very chungus log but it happened at the end of it. The world's join notifier went off and not even a second later. Died.

AdmiralSoap commented 2 years ago

DESKTOP-M3LTEEK - 2021.10.28.1280 - 2021-10-30 02_56_13.log

I was hosting a session and suddenly majority of the people in the session froze. Their audio was still working, but they were forced to crash Neos. Only me and one other person survived with no issues.

orange3134 commented 2 years ago

Recently, I often experience a sudden disconnection from a session. I'm not sure if it's related to this issue, but I'll provide a log when it happens. In this log, I attended a friend-hosted MEROOM session at 13:48 and was suddenly disconnected after about a minute SHIRUKO2 - 2021.10.28.1280 - 2021-10-30 13_06_56.log .

orange3134 commented 2 years ago

I ran into the same problem that Admiral Soap reported. Several people in the session froze. I could only hear the voice. They had to restart NeosVR.

Here is the log provided by one of them.

DESKTOP-6KCBS27_-2021.10.30.605-_2021-10-30_19_29_52.log

Nordwig commented 2 years ago

I had something simmilar a moment ago. I was hosting a world with about 10 players. Every thing worked without any problems for about 2 hours. Then suddenly my view was frozen for about 20sec and I was able to hear everyone and communicate without any problems while frozen. Then the updating worked for like 2sec then I was frozen again. All users stayed in the session and got the updates when I wasn't frozen. This looped for about 5 - 10 min until I decided to close Neos with the [X] on the NeosVR window. The Session closed immediately and i was thrown into the closing screen like normal. Closing took a bit longer then usual but without a crash.

I restarted Neos with out closing anything else. Now I got the same looping effect alone in my Home world. latest Build 2021.10.30.605

DESKTOP-FKE488C - 2021.10.30.605 - 2021-10-30 14_14_35.log

Player.log

Additional: Restarting Neos : problem was still there Restarting SteamVR + Neos : problem was still there Restarting Steam + SteamVR + Neos : problem was still there Disconnected Vive Face Tracker and restarted Vive SR-Runtime & Service + Steam + SteamVR + Neos: problem solved Reconnecting Vive Face Tracker and restart Neos: Problem solved an face tracker working. (System was not restartet at all) Hope that helps

orange3134 commented 2 years ago

I got the log of the headless server when the phenomenon that seems to be related to this occurred.

In the first case, each user was able to move around and have a conversation, but the position of some objects seemed to be out of sync. The position of the objects as seen by each user did not match. LogiX, which requires a Host user, didn't seem to be working. SUNAGIMO - 2021.10.30.605 - 2021-10-30 22_04_30.log

In the second case, most users got stuck. The conversation was possible. The headless server no longer accepts console commands. SUNAGIMO - 2021.10.30.605 - 2021-10-30 22_53_00.log

BlueCyro commented 2 years ago

Several other users have had this happen to them in the session I was in multiple times. Here is my log: DESKTOP-N6MFPU0 - 2021.10.30.605 - 2021-11-03 05_50_31.log

Frooxius commented 2 years ago

Thanks for the info! Unfortunately there's not too much info the logs that I can find first, I'll have to dig through them a bit more.

However I added a mechanism to get some more diagnostic data in 2021.11.3.684 (available through the custom installer for now). If you can switch to that build that could help with this!

If you freeze, please leave it running for at last a minute before shutting down, so it gathers enough data.

AshtonSparx commented 2 years ago

Will do this now! Thanks Frooxius!

For others, Please refrain from sharing any further logs if you're on the Steam Build until further notice. Thanks guys!

orange3134 commented 2 years ago

My friend ran into a similar problem. There was a lot of people attending the event, one of which was using the latest build, so they provided the logs.

She said the screen froze and she could only hear the audio. She also said that the other participants froze at the same time as her. It seems to have happened on both headless clients and regular client-hosted sessions.

DESKTOP-6UKEJ2J_-2021.11.3.684-_2021-11-03_21_05_06.log

DESKTOP-6UKEJ2J_-2021.11.3.684-_2021-11-03_21_34_24.log

Log provider: K-Pere (The second log may not be relevant as far as I can see, but she provided me with these two logs so I'll post them as is.)

orange3134 commented 2 years ago

This is a log of the latest build with the same issue as this reply. https://github.com/Neos-Metaverse/NeosPublic/issues/3265#issuecomment-955148725

It doesn't freeze, but suddenly disconnects from the session. No error message is displayed on Neos. The log shows a timeout. The session is ongoing and only some users will be disconnected.

SHIRUKO2 - 2021.11.3.684 - 2021-11-03 22_43_54.log

shiftyscales commented 2 years ago

I had this happen tonight right as a friend joined my private session (cloud home.) SHIFTYWINDOWS - 2021.11.3.684 - 2021-11-03 13_14_54.log

Frooxius commented 2 years ago

Thank you for the logs! This actually helps narrow it down quite a lot.

@shiftyscales What happened near the end? How long did you leave Neos running? The log just ends abruptly, right about the time it tries to capture the stack trace of the stuck thread. Did Neos crash? Did it stay frozen?

shiftyscales commented 2 years ago

It stayed waiting at the Steam 'waiting' screen for about five or so minutes I'd estimate before I closed it. If it happens again, I guess I'll just leave it considerably longer, @Frooxius.

AlexThorin commented 2 years ago

Few new logs for you guys using the latest neos website build as of right now ( Beta 2020.9.3.357 )

Player-prev.log FOXSTATION - 2021.11.3.684 - 2021-11-03 08_17_09.log

Left it open till my computer was nearly unresponsive. For some reason it seems to drop performance across my entire pc, though i'm unsure if this is an issue specific to my setup, since i havent seen anyone else report this. Maybe 5 or so minutes. Didn't think to look at the time unfortunately.

Frooxius commented 2 years ago

I've pushed a new build 2021.11.4.402 which adds some extra diagnostic information to help to further narrow this down. Can you send me logs from this build please when this occurs again?

shiftyscales commented 2 years ago

SHIFTYWINDOWS - 2021.11.4.402 - 2021-11-04 11_33_39.log

Frooxius commented 2 years ago

Getting closer! I added more diagnostics in 2021.11.5.486 as well as potential fix (it's based on a hunch through, rather than hard data).

Please send me logs from that build if you encounter it again. I'll probably ask you to send another file as well based on the log (it dumps the problematic network message so I can analyze it later).

AdmiralSoap commented 2 years ago

DESKTOP-M3LTEEK - 2021.11.5.486 - 2021-11-06 01_44_32.log just happened to me. froze and everyone got kicked out

AshtonSparx commented 2 years ago

DESKTOP-M3LTEEK - 2021.11.5.486 - 2021-11-06 01_44_32.log just happened to me. froze and everyone got kicked out

For Froox, relevant lines seems to be 6144 and onward

orange3134 commented 2 years ago

My friend provided a new log and I will post it on her behalf. Log provider: K-Pere

She described the situation as follows: The synchronization is lost, the audio begins to be interrupted, and the audio is completely interrupted in about 3 minutes. Then she is returned to Home.

DESKTOP-6UKEJ2J_-2021.11.5.486-_2021-11-06_15_06_15.log

orange3134 commented 2 years ago

Another user has sent me a log about this issue.

He is often suddenly disconnected from the session and returned to his home world. Log provider: ShoeiP

MYCOMPUTER - 2021.11.5.486 - 2021-11-05 21_04_08.log MYCOMPUTER - 2021.11.5.486 - 2021-11-05 21_07_16.log MYCOMPUTER - 2021.11.5.486 - 2021-11-06 21_13_31.log

Frooxius commented 2 years ago

Hmm. These seem like different issue. The original would always only happen to people in the world, but never the host it seemed.

AshtonSparx commented 2 years ago

Hmm. These seem like different issue. The original would always only happen to people in the world, but never the host it seemed.

Any good news with the prior logs (Not submitted by orange) that Soap submitted?

bontebok commented 2 years ago

Just an update from my end. After discovering that I was behind a Type-3 NAT due to my OPNsense (pfSense based) router and making subsequent changes to provide a Moderate Type-2 NAT (Static Port, Open, etc), all of my freezes in Neos have gone away. I was freezing up in worlds 2-6 times per session and I can report that I have not froze up a single time since making that change this week.

I encourage those who are having Steam "Waiting..." freezes on their client to ensure that they are connecting directly to other users and not going through the LNL Relay.

I updated the Networking Information (https://wiki.neos.com/Networking_Information/en) entry on the Wiki for more information about Neos networking and how to determine if you are going direct (peer to peer) or if you are going through the LNL Relay.

Enverex commented 2 years ago

This happened again today and ruined an important event. The log is also a pain to look through due to another issue that has yet to be resolved (https://github.com/Neos-Metaverse/NeosPublic/issues/2934) which has bloated the log beyond all reasonable levels.

Full log of the session that basically just... stopped. lockup1.log

orange3134 commented 2 years ago

Log with the latest version. I connected to the "アバターセットアップをやるw" Session twice, but was disconnected in a few minutes. SHIRUKO2 - 2021.11.7.784 - 2021-11-08 22_53_24.log

RyuviTheViali commented 2 years ago

RYUVI-TOWER - 2021.11.7.784 - 2021-11-08 20_54_52.log

Here is a log, client version 2021.11.7.784 Non-steam version. I was the host of the world I was in - Mad Max spawned a short 3-second video which fully loaded and started to play on my end, but shortly after Max deleted the video. Upon deleting the video, my client froze.

AshtonSparx commented 2 years ago

Yesterday, June was experiencing the Neos hanging very frequently during the same session. I had an audio stream playing in the world. When I removed the stream, his client wasn't hanging anymore during the rest of our time.

Here's his logs he's sent me. They're all very small logs because he'd freeze after a few minutes from joining.

DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_17_30_34.log DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_11_52.log DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_24_21.log DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_38_35.log

orange3134 commented 2 years ago

This is the headless log when the problem occurred in version 2021.11.7.1245. Around 23:00, U-toritori and U-Regale tried to connect to the headless many times, but they were disconnected immediately after connecting. They are not frozen, they are simply disconnected. DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_38_35.log

By the way, should I create another issue for disconnecting issues without client freezes?

Frooxius commented 2 years ago

Thank you all for the logs!

Hmm, it's looking that it's actually something else from what I initially suspected though, the bug is now looking different from others. I've added some more diagnostics in 2021.11.10.899, hopefully they'll help narrow this down.

@RyuviTheViali Can you send me this file from your system please? C:/Users/Ryuvi/AppData/Local/Temp/Solirax/NeosVR\Cache\rnmkcbsh2ztmxuzv Or any of the others from lines in the log that start with Dumping current sync message to:

Lexevolution commented 2 years ago

I just got kicked to my local home while not doing much, here's the log: LEXDESKTOP - 2021.11.10.1265 - 2021-11-11 10_52_06.log

Nordwig commented 2 years ago

Neos froze with "Waiting for Neos" then shut down after around 30 sec. DESKTOP-FKE488C - 2021.11.10.1253 - 2021-11-11 16_31_58.log

Gawdl3y commented 2 years ago

Had a world crash last night, probably not the same issue (appears to be physics-related, happened right around a user joining), but wanted to make the log available. Relevant exception is at 11:44:50 PM.548.

SCHUYLER-DT - 2021.11.10.1253 - 2021-11-10 21_25_03.log

Nordwig commented 2 years ago

I hosted the Session, Neos froze "Waiting for Neos" after a few seconds new frames where displayed and Neos closed.

[error.log](https://github.com/Neos-Metaverse/NeosPublic/files/7548924/error.log Player.log DESKTOP-FKE488C - 2021.11.10.1253 - 2021-11-16 18_02_33.log ) crash.zip

shadowpanther commented 2 years ago

Was working on an avatar, grabbed glasses, froze. Log shows nansplosion. Might be because of virtualparent with invalid slot target. VIXEN - 2021.11.10.1265 - 2021-11-16 22_34_11.log

AlexThorin commented 2 years ago

Think the issue slightly evolved to something, different for me. Not really better or worse. Just kinda freezes the view and nothing moved, but still entirely connected. Looking around just shows the frozen image, but without the steam waiting grid. I did also check to confirm i had fade to grid on app hang turned on in steam vr. Whatever's happening, steam doesn't think the app hung. The game wont crash on it's own, but attempting to close it with the X will cause a unity crash.

FOXSTATION - 2021.11.10.1265 - 2021-12-07 16_24_52.log Player.log

Enverex commented 2 years ago

Happened on my headless again overnight. Neos becomes a zombie process. All session lists still show the sessions as being there, but the headless process has become unresponsive. Log attached.

q3 - 2021.11.10.1253 - 2021-12-22 03_15_48.log.txt

The only thing out of the ordinary in there is a bunch of errors regarding an external image (which I assume was on someone's avatar) not loading, specifically - https://cdn3.iconfinder.com/data/icons/popular-services-brands-vol-2/512/discord-512.png

AshtonSparx commented 2 years ago

The only thing out of the ordinary in there is a bunch of errors regarding an external image (which I assume was on someone's avatar) not loading, specifically - https://cdn3.iconfinder.com/data/icons/popular-services-brands-vol-2/512/discord-512.png

9/10 that's from someone's injected "badge" lol

Neos-Metaverse / NeosPublic