Open AshtonSparx opened 2 years ago
I see one issue in one of the logs so far, but not much else. I'll try to fix that one up, that should help some.
Some of the logs you posted failed to upload
I see one issue in one of the logs so far, but not much else. I'll try to fix that one up, that should help some.
Some of the logs you posted failed to upload
* How do the crashes manifest exactly? Is it always a world crash? Or something else? * Did your headless crash too? * Does this happen specifically in the BLFC world and not any other?
Which logs failed to upload? I can try again or upload to a hidden GH file thingy.
For the other questions
How do the crashes manifest exactly? Is it always a world crash? Or something else?
The world crashes instantly kicks the user from the world and sends them to their last focused world. The game doesn't actually full crash. It'll just forever stay not responding. If you're in VR. It just stays on "Waiting on Neos"
My headless doesn't seem effected by it. It just stays open as normal.
This happened in other worlds, not just headless's.
@AshtonSparx The links for your own user logs seem to be a bit garbled.
What's the timeline on the crash? If I understand it, they get kicked into last focused world and then Neos freezes? Is there a delay between getting kicked out and freezing?
From what I've witnessed personally, the getting kicked and freezing are separate from each other.
The freezing seems to be like Neos having a blindfold and shotgunning into a crowd. In the BLFC world. A handful of users would all at the same time freeze out of no where. Seems to possibly be tied to users joining/leaving. From what Epsilion was gathering. It wouldn't always be the same users, but other users also wouldn't freeze even if others did several times during the same session.
Contributing my logs to this. Interestingly, mine might have been something else, since it was locking up and slowing down the rest of my computer too even though it was at 50% cpu and had about 6 GB of ram free. Once neos was closed everything went back to normal. Not sure if neos just started DUMPING stuff into logs at a super high rate? Dunno how to read logs though so i'll just let you guys figure it out
Player.log FOXSTATION - 2021.10.26.9 - 2021-10-28 02_46_06.log
Feels like the same outcome though. Can still talk and are still connected, but everything's frozen
Here are my logs, however I can not find any concrete relationship between when I would crash and what was happening in the world. The world would just hang in the current session on the "waiting" screen but you could freely talk with the individuals until you decided to close out of the session.
I've added an exception guard and some extra diagnostics in 2021.10.28.1280. Can you please test with that build and see if there are any improvements?
Please share any new logs if you run into issues (even if it doesn't fully crash).
I've added an exception guard and some extra diagnostics in 2021.10.28.1280. Can you please test with that build and see if there are any improvements?
Please share any new logs if you run into issues (even if it doesn't fully crash).
Will do this immediately! We'll crunch this mystery together! Thanks Mr. Froox! Will keep you updated.
I too can report I am having this issue. It happens on average about once every 2 hours. Another user said it may be my Vive Face Tracker as that hardware is not typical for many users. I can report that after about 3 days without the face tracker plugged in, I am still experiencing Steam Waiting... hangs.
One thing to note, the users can still hear me in the world and I can hear them however no display updates occur. Having the host kick me from the world still leaves me "in" the world, meaning I still hear ambient audio playing but users can't hear me and I cannot hear them.
If there's anything I can add to be helpful to diagnosing this, let me know. Rucio#0134 on Discord.
I too can report I am having this issue. It happens on average about once every 2 hours. Another user said it may be my Vive Face Tracker as that hardware is not typical for many users. I can report that after about 3 days without the face tracker plugged in, I am still experiencing Steam Waiting... hangs.
One thing to note, the users can still hear me in the world and I can hear them however no display updates occur. Having the host kick me from the world still leaves me "in" the world, meaning I still hear ambient audio playing but users can't hear me and I cannot hear them.
If there's anything I can add to be helpful to diagnosing this, let me know. Rucio#134 on Discord.
Is this still happening on Neos Version 2021.10.28.1280
?
Is this still happening on Neos Version
2021.10.28.1280
?
It was, but I think I may have resolved it as I was able to be in world for at least an hour and a half without a crash, however I need to do more testing to see if I am completely void of the problem.
The steps I took were to completely uninstall SteamVR and Neos, insuring that I deleted the AppData\LocalLow\Solirax\NeosVR folder which included the Neos.lightdb and Assets folder. Once I reinstalled SteamVR and Neos, I had to login and run through the tutorial but after that things seemed to have improved.
I'll report back tomorrow my results.
Confirmation, it still exists. The moment someone joined the world, everyone's Neos hung except for the host. DESKTOP-SPEUN1P - 2021.10.28.1280 - 2021-10-28 17_46_55.log
This is a very chungus log but it happened at the end of it. The world's join notifier went off and not even a second later. Died.
DESKTOP-M3LTEEK - 2021.10.28.1280 - 2021-10-30 02_56_13.log
I was hosting a session and suddenly majority of the people in the session froze. Their audio was still working, but they were forced to crash Neos. Only me and one other person survived with no issues.
Recently, I often experience a sudden disconnection from a session. I'm not sure if it's related to this issue, but I'll provide a log when it happens. In this log, I attended a friend-hosted MEROOM session at 13:48 and was suddenly disconnected after about a minute SHIRUKO2 - 2021.10.28.1280 - 2021-10-30 13_06_56.log .
I ran into the same problem that Admiral Soap reported. Several people in the session froze. I could only hear the voice. They had to restart NeosVR.
Here is the log provided by one of them.
I had something simmilar a moment ago. I was hosting a world with about 10 players. Every thing worked without any problems for about 2 hours. Then suddenly my view was frozen for about 20sec and I was able to hear everyone and communicate without any problems while frozen. Then the updating worked for like 2sec then I was frozen again. All users stayed in the session and got the updates when I wasn't frozen. This looped for about 5 - 10 min until I decided to close Neos with the [X] on the NeosVR window. The Session closed immediately and i was thrown into the closing screen like normal. Closing took a bit longer then usual but without a crash.
I restarted Neos with out closing anything else. Now I got the same looping effect alone in my Home world. latest Build 2021.10.30.605
DESKTOP-FKE488C - 2021.10.30.605 - 2021-10-30 14_14_35.log
Additional: Restarting Neos : problem was still there Restarting SteamVR + Neos : problem was still there Restarting Steam + SteamVR + Neos : problem was still there Disconnected Vive Face Tracker and restarted Vive SR-Runtime & Service + Steam + SteamVR + Neos: problem solved Reconnecting Vive Face Tracker and restart Neos: Problem solved an face tracker working. (System was not restartet at all) Hope that helps
I got the log of the headless server when the phenomenon that seems to be related to this occurred.
In the first case, each user was able to move around and have a conversation, but the position of some objects seemed to be out of sync. The position of the objects as seen by each user did not match. LogiX, which requires a Host user, didn't seem to be working. SUNAGIMO - 2021.10.30.605 - 2021-10-30 22_04_30.log
In the second case, most users got stuck. The conversation was possible. The headless server no longer accepts console commands. SUNAGIMO - 2021.10.30.605 - 2021-10-30 22_53_00.log
Several other users have had this happen to them in the session I was in multiple times. Here is my log: DESKTOP-N6MFPU0 - 2021.10.30.605 - 2021-11-03 05_50_31.log
Thanks for the info! Unfortunately there's not too much info the logs that I can find first, I'll have to dig through them a bit more.
However I added a mechanism to get some more diagnostic data in 2021.11.3.684 (available through the custom installer for now). If you can switch to that build that could help with this!
If you freeze, please leave it running for at last a minute before shutting down, so it gathers enough data.
Will do this now! Thanks Frooxius!
For others, Please refrain from sharing any further logs if you're on the Steam Build until further notice. Thanks guys!
My friend ran into a similar problem. There was a lot of people attending the event, one of which was using the latest build, so they provided the logs.
She said the screen froze and she could only hear the audio. She also said that the other participants froze at the same time as her. It seems to have happened on both headless clients and regular client-hosted sessions.
DESKTOP-6UKEJ2J_-2021.11.3.684-_2021-11-03_21_05_06.log
DESKTOP-6UKEJ2J_-2021.11.3.684-_2021-11-03_21_34_24.log
Log provider: K-Pere (The second log may not be relevant as far as I can see, but she provided me with these two logs so I'll post them as is.)
This is a log of the latest build with the same issue as this reply. https://github.com/Neos-Metaverse/NeosPublic/issues/3265#issuecomment-955148725
It doesn't freeze, but suddenly disconnects from the session. No error message is displayed on Neos. The log shows a timeout. The session is ongoing and only some users will be disconnected.
I had this happen tonight right as a friend joined my private session (cloud home.) SHIFTYWINDOWS - 2021.11.3.684 - 2021-11-03 13_14_54.log
Thank you for the logs! This actually helps narrow it down quite a lot.
@shiftyscales What happened near the end? How long did you leave Neos running? The log just ends abruptly, right about the time it tries to capture the stack trace of the stuck thread. Did Neos crash? Did it stay frozen?
It stayed waiting at the Steam 'waiting' screen for about five or so minutes I'd estimate before I closed it. If it happens again, I guess I'll just leave it considerably longer, @Frooxius.
Few new logs for you guys using the latest neos website build as of right now ( Beta 2020.9.3.357 )
Player-prev.log FOXSTATION - 2021.11.3.684 - 2021-11-03 08_17_09.log
Left it open till my computer was nearly unresponsive. For some reason it seems to drop performance across my entire pc, though i'm unsure if this is an issue specific to my setup, since i havent seen anyone else report this. Maybe 5 or so minutes. Didn't think to look at the time unfortunately.
I've pushed a new build 2021.11.4.402 which adds some extra diagnostic information to help to further narrow this down. Can you send me logs from this build please when this occurs again?
Getting closer! I added more diagnostics in 2021.11.5.486 as well as potential fix (it's based on a hunch through, rather than hard data).
Please send me logs from that build if you encounter it again. I'll probably ask you to send another file as well based on the log (it dumps the problematic network message so I can analyze it later).
DESKTOP-M3LTEEK - 2021.11.5.486 - 2021-11-06 01_44_32.log just happened to me. froze and everyone got kicked out
DESKTOP-M3LTEEK - 2021.11.5.486 - 2021-11-06 01_44_32.log just happened to me. froze and everyone got kicked out
For Froox, relevant lines seems to be 6144 and onward
My friend provided a new log and I will post it on her behalf. Log provider: K-Pere
She described the situation as follows: The synchronization is lost, the audio begins to be interrupted, and the audio is completely interrupted in about 3 minutes. Then she is returned to Home.
Another user has sent me a log about this issue.
He is often suddenly disconnected from the session and returned to his home world. Log provider: ShoeiP
MYCOMPUTER - 2021.11.5.486 - 2021-11-05 21_04_08.log MYCOMPUTER - 2021.11.5.486 - 2021-11-05 21_07_16.log MYCOMPUTER - 2021.11.5.486 - 2021-11-06 21_13_31.log
Hmm. These seem like different issue. The original would always only happen to people in the world, but never the host it seemed.
Hmm. These seem like different issue. The original would always only happen to people in the world, but never the host it seemed.
Any good news with the prior logs (Not submitted by orange) that Soap submitted?
Just an update from my end. After discovering that I was behind a Type-3 NAT due to my OPNsense (pfSense based) router and making subsequent changes to provide a Moderate Type-2 NAT (Static Port, Open, etc), all of my freezes in Neos have gone away. I was freezing up in worlds 2-6 times per session and I can report that I have not froze up a single time since making that change this week.
I encourage those who are having Steam "Waiting..." freezes on their client to ensure that they are connecting directly to other users and not going through the LNL Relay.
I updated the Networking Information (https://wiki.neos.com/Networking_Information/en) entry on the Wiki for more information about Neos networking and how to determine if you are going direct (peer to peer) or if you are going through the LNL Relay.
This happened again today and ruined an important event. The log is also a pain to look through due to another issue that has yet to be resolved (https://github.com/Neos-Metaverse/NeosPublic/issues/2934) which has bloated the log beyond all reasonable levels.
Full log of the session that basically just... stopped. lockup1.log
Log with the latest version. I connected to the "アバターセットアップをやるw" Session twice, but was disconnected in a few minutes. SHIRUKO2 - 2021.11.7.784 - 2021-11-08 22_53_24.log
RYUVI-TOWER - 2021.11.7.784 - 2021-11-08 20_54_52.log
Here is a log, client version 2021.11.7.784 Non-steam version. I was the host of the world I was in - Mad Max spawned a short 3-second video which fully loaded and started to play on my end, but shortly after Max deleted the video. Upon deleting the video, my client froze.
Yesterday, June was experiencing the Neos hanging very frequently during the same session. I had an audio stream playing in the world. When I removed the stream, his client wasn't hanging anymore during the rest of our time.
Here's his logs he's sent me. They're all very small logs because he'd freeze after a few minutes from joining.
DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_17_30_34.log DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_11_52.log DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_24_21.log DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_38_35.log
This is the headless log when the problem occurred in version 2021.11.7.1245. Around 23:00, U-toritori and U-Regale tried to connect to the headless many times, but they were disconnected immediately after connecting. They are not frozen, they are simply disconnected. DESKTOP-E9C0718_-2021.11.7.794-_2021-11-08_18_38_35.log
By the way, should I create another issue for disconnecting issues without client freezes?
Thank you all for the logs!
Hmm, it's looking that it's actually something else from what I initially suspected though, the bug is now looking different from others. I've added some more diagnostics in 2021.11.10.899, hopefully they'll help narrow this down.
@RyuviTheViali Can you send me this file from your system please? C:/Users/Ryuvi/AppData/Local/Temp/Solirax/NeosVR\Cache\rnmkcbsh2ztmxuzv
Or any of the others from lines in the log that start with Dumping current sync message to:
I just got kicked to my local home while not doing much, here's the log: LEXDESKTOP - 2021.11.10.1265 - 2021-11-11 10_52_06.log
Neos froze with "Waiting for Neos" then shut down after around 30 sec. DESKTOP-FKE488C - 2021.11.10.1253 - 2021-11-11 16_31_58.log
Had a world crash last night, probably not the same issue (appears to be physics-related, happened right around a user joining), but wanted to make the log available. Relevant exception is at 11:44:50 PM.548.
I hosted the Session, Neos froze "Waiting for Neos" after a few seconds new frames where displayed and Neos closed.
[error.log](https://github.com/Neos-Metaverse/NeosPublic/files/7548924/error.log Player.log DESKTOP-FKE488C - 2021.11.10.1253 - 2021-11-16 18_02_33.log ) crash.zip
Was working on an avatar, grabbed glasses, froze. Log shows nansplosion. Might be because of virtualparent with invalid slot target. VIXEN - 2021.11.10.1265 - 2021-11-16 22_34_11.log
Think the issue slightly evolved to something, different for me. Not really better or worse. Just kinda freezes the view and nothing moved, but still entirely connected. Looking around just shows the frozen image, but without the steam waiting grid. I did also check to confirm i had fade to grid on app hang turned on in steam vr. Whatever's happening, steam doesn't think the app hung. The game wont crash on it's own, but attempting to close it with the X will cause a unity crash.
FOXSTATION - 2021.11.10.1265 - 2021-12-07 16_24_52.log Player.log
Happened on my headless again overnight. Neos becomes a zombie process. All session lists still show the sessions as being there, but the headless process has become unresponsive. Log attached.
q3 - 2021.11.10.1253 - 2021-12-22 03_15_48.log.txt
The only thing out of the ordinary in there is a bunch of errors regarding an external image (which I assume was on someone's avatar) not loading, specifically - https://cdn3.iconfinder.com/data/icons/popular-services-brands-vol-2/512/discord-512.png
The only thing out of the ordinary in there is a bunch of errors regarding an external image (which I assume was on someone's avatar) not loading, specifically - https://cdn3.iconfinder.com/data/icons/popular-services-brands-vol-2/512/discord-512.png
9/10 that's from someone's injected "badge" lol
Describe the bug?
Neos BLFC world, bunch of user's neos starts hanging but not for everyone. On my headless it's happened as well. I'll be supplying everyone's log that was present. We're not 100% certain what exactly is causing it. There doesn't seem to be a common occurance. I did notice twice when I went into free cam mode, everyone instantly got a world crash and had to rejoin.
Relevant issues
Possibly #3244 ??
To Reproduce
Like I mentioned before, I have no idea how to reproduce hence supplying a bunch of user's logs.
Expected behavior
For Neos to not hang without a crash error.
Log Files
Mine headless's Logs: DESKTOP-SPEUN1P - 2021.10.26.9 - 2021-10-25 21_14_10.log
My User Log:
[DESKTOP-SPEUN1P - 2021.10.26.9 - 2021-10-28 00_2700.log](https://github.com/Neos-Metaverse/NeosPublic/files/7431892/DESKTOP-SPEUN1P.-.2021.10.26.9.-.2021-10-28.00
DESKTOP-SPEUN1P - 2021.10.26.9 - 2021-10-28 02_47_08.log 27_00.log)
My Player Logs: Player.log Player-prev.log
My logs when my game hung in BLFC pool world: Player.log DESKTOP-SPEUN1P_-2021.10.26.9-_2021-10-27_21_12_08.log
Soap's Log:
DESKTOP-M3LTEEK_-2021.10.26.9-_2021-10-27_21_03_13.log
Screenshots
No response
How often does it happen?
Sometimes
Does the bug persist after restarting Neos?
Yes
Neos Version Number
2021.10.26.9
What Platforms does this occur on?
Windows
Link to Reproduction Item/World
No response
Did this work before?
No
If it worked before, on which build?
No response
Additional context
If anyone else has log files of when Neos hangs and doesn't return or you suddenly get a world crash error. Please contribute some logs!
Your player log can be found:
\AppData\LocalLow\Solirax\NeosVR
Your normal log can be found:\steamapps\common\NeosVR\Logs
Reporters
AshtonSparx#5050 AdmiralSoap#4934