secondlife / jira-archive

2 stars 0 forks source link

[BUG-230568] Music Dropping in world longer then the typical 15 seconds all over SL, it affects all streams and radio stations. #8214

Closed sl-service-account closed 7 months ago

sl-service-account commented 3 years ago

What just happened?

After gather information from Djs, and Singers around secondlife, the music in world starts dropping longer then the typical 15 second delay around 3Pm and the drop time gets long as the evening goes on. The worst times for the music drop is 4-6PM. The time drop can range from 1 minute - 3 minutes. It affects all areas equally, The issue is now morphing into not only the stream dropping in world but the music turning off completely. There is a period of time we have to wait to turn music in world back on. This happens constantly at this point.

The issues does not affect everyone at the location at the same time, it varies. If I'm in a club with ten other people the music can drop for three of us, and not the other seven, where as the reverse can also be true.

The times we have the most issues between 3 Pm sLt and 8 PM slt is when the American and Canadian users log online and the most bandwidth is used in North America, We had thought it could be where the shoutcast and icecast servers are from but seeing as their also in the US with the exception of Vside Radio who sells their streams on private servers, this cause was debunked. 

What were you doing when it happened?

Djing and collecting information, most people were managing clubs at the time.

What were you expecting to happen instead?

Music usually reconects after 10-15 seconds. this is no longer the case.

Other information

I am still recieving sim information from SL CLients

Attachments

Links

Related

Original Jira Fields | Field | Value | | ------------- | ------------- | | Issue | BUG-230568 | | Summary | Music Dropping in world longer then the typical 15 seconds all over SL, it affects all streams and radio stations. | | Type | Bug | | Priority | Unset | | Status | Closed | | Resolution | Triaged | | Reporter | Destimona (destimona) | | Assignee | Dan Linden (dan.linden) | | Created at | 2021-04-13T23:52:03Z | | Updated at | 2022-07-20T17:22:13Z | ``` { 'Build Id': 'unset', 'Business Unit': ['Platform'], 'Date of First Response': '2021-04-19T09:01:57.928-0500', "Is there anything you'd like to add?": 'I am still recieving sim information from SL CLients', 'ReOpened Count': 0.0, 'Severity': 'Unset', 'System': 'SL Viewer', 'Target Viewer Version': 'viewer-development', 'What just happened?': 'After gather information from Djs, and Singers around secondlife, the music in world starts dropping longer then the typical 15 second delay around 3Pm and the drop time gets long as the evening goes on. The worst times for the music drop is 4-6PM. The time drop can range from 1 minute - 3 minutes. It affects all areas equally,', 'What were you doing when it happened?': 'Djing and collecting information, most people were managing clubs at the time.', 'What were you expecting to happen instead?': 'Music usually reconects after 10-15 seconds. this is no longer the case.', 'Where': 'All Grids', } ```
sl-service-account commented 3 years ago

Dan Linden commented at 2021-04-19T14:01:58Z

Hi Destimona,

Thank you for the report. You mention a "typical 15 second delay". This is a delay of what?

How and where can we see this issue?

When filing a Jira please provide more details of your problem. Without this information your jira will be closed as Needs More information and will not be triaged.

Please include:

(!) Which version of the client are you using? (This can be found in Help, About Second Life)

(!) Your computer specs (from Help, About Second Life)

(!) A description of the problem, explaining what you do / see / don't see, including any error messages displayed

(!) Step by step Directions on how to reproduce your experience.

(!) A screenshot of what you are seeing. (Perhaps a video in this case to capture what you are hearing.)

Please press the Information Provided button when you have added the additional information. Thanks!

sl-service-account commented 3 years ago

Destimona commented at 2021-04-20T00:55:07Z

In secondlife viewer and firestorm the music in world is dropping, usually there is a slight delay in sound that lasts 15 seconds. You can easy replicate this issue by going to any sim and listening to the music in world between 3 pm slt and 9 pm slt as stated above, the music will stop playing.

sl-service-account commented 3 years ago

DJxScotspaul commented at 2021-04-20T13:25:42Z

Dan sorry but read the Jira before replying !

See this is why nothing gets fixed because you dont do any research

We have been complaining via tickets for MONTHS

I PERSONALLY collected loads of data from DJs around the grid, its not just us its 1000s of ppl affected

Now i got a reply it was an FMOD issue with FS viewer

But thanks for your reply

sl-service-account commented 3 years ago

Dan Linden commented at 2021-04-20T17:50:15Z

Hi Destimona,

sl-service-account commented 3 years ago

Marianne McCann commented at 2021-04-25T21:30:06Z

I've seen similar happening, but not with a 15 minute specific time frame that I've noted.

Still after a period of time of listening to a stream – any stream: live DJ, internet radio station, etc. – the stream will turn itself off, requiring me to re-click play to pick up the music. 

The stream itself is fine, but the viewer just decides to drop it. 

This has been the case for me for ~1 year, on multiple Mac clients, but I've heard similar from Windows users. 

sl-service-account commented 3 years ago

Dan Linden commented at 2021-04-27T16:50:23Z

Hi @djxscotspaul As you have put lots of effort into collecting data, you may be best able to help describe the problem and provide clues to the cause of the issue. Does the delay described in the description mean the amount of time between the DJ speaking and the audience hearing it in SL? Please provide specific places (region and coordinates) where this bug can be heard. Which streaming providers are being used at these areas?

Thank you, Dan

sl-service-account commented 3 years ago

DJxScotspaul commented at 2021-04-27T17:06:45Z

Hi Dan

You can find the ticket here that was started months ago and is ongoing

https://lindenlab.freshdesk.com/support/tickets/1619675

I have also included a response from Pantera that might help you [!https://yt3.ggpht.com/ytc/AAUvwnjLVl8ROcdS5NcjdeMjXLdo6sLCzmEjdaZByv37Uw=s48-c-k-c0x00ffffff-no-rj\|width=40,height=40,id=img!|https://www.youtube.com/channel/UChturH5PPNoRhlSXf6I_YEQ]

 

Pantera Północy2 months ago (edited) I've got an info from another FS team member, that FMOD team (the one, that makes sound sub-module) "have found that since supporting looping of http streams, a malformed parameter got into the seek command, so it returns that error on some server setups (if it is configured that way). " And that they've "updated the code to be fixed in the next release on all major versions of fmod api". It is unlikely that this FMOD update would get into the upcoming version of Firestorm, but the next one after the upcoming one - most probably so. Right now Firestorm is using the latest possible version, named 2.01.08 - https://fmod.com/resources/documentation-studio?version=2.1&page=welcome-to-fmod-studio-revision-history.html - you can track the changelog yourself and see when the bug mentioned above gets fixed in the library itself.Show less     REPLY     [!https://yt3.ggpht.com/ytc/AAUvwnjLVl8ROcdS5NcjdeMjXLdo6sLCzmEjdaZByv37Uw=s48-c-k-c0x00ffffff-no-rj\|width=40,height=40,id=img!|https://www.youtube.com/channel/UChturH5PPNoRhlSXf6I_YEQ] Highlighted reply

 

Pantera Północy1 week ago The 2.01.09 version was released, you can check for yourself if it was fixed: https://fmod.com/resources/documentation-api?version=2.1&page=welcome-revision-history.html

sl-service-account commented 3 years ago

Dan Linden commented at 2021-04-27T17:21:25Z

Thank you for the information, DJxScotpaul,

Please tell us which viewer and version you are using, even if it's not Second Life. And include the Audio Driver Version from Help > About (Second Life or whatever)

The Second Life viewer uses FMOD Studio 2.00.11, so this bug may not reproduce there. Are you able to reproduce this bug with the Second Life viewer?

Thank you, Dan

sl-service-account commented 3 years ago

Destimona commented at 2021-04-27T17:34:41Z

We wondered about this so Scotspaul did try with the second life viewer it was no different to the Firestorm viewer. Im using the newest firestorm 

Firestorm 6.4.13 (63251) Mar 2 2021 11:39:37 (32bit / SSE2) (Firestorm-Release) with Havok support

Settings mode: Firestorm Viewer Skin: Firestorm (Grey) Window size: 1366x705 px Font Used: Deja Vu (96 dpi) Font Size Adjustment: 0 pt UI Scaling: 1 Draw distance: 128 m Bandwidth: 1500 kbit/s LOD factor: 2 Render quality: Medium (3/7) Advanced Lighting Model: No Texture memory: 512 MB (1) VFS (cache) creation time (UTC): 2021-4-24T0:26:27 Built with MSVC version 1916 Packets Lost: 441/40,333 (1.1%) April 27 2021 10:32:59 SLT

 

The reason I couldn't specify  a location was because it's so broad spread around the grid I'd end up just sending slurls to every sim 

sl-service-account commented 3 years ago

JIRAUSER333555 commented at 2021-04-28T20:34:57Z

Just cutting out now and buggering up our set @ subculture - not good

sl-service-account commented 3 years ago

Dan Linden commented at 2021-04-29T23:08:27Z

Subculture at http://maps.secondlife.com/secondlife/Atlantis%20City/237/166/23 is using http://nebula.shoutca.st:8094 , SHOUTcast Server v2.5.5.733/posix(linux x64)

sl-service-account commented 3 years ago

Dan Linden commented at 2021-04-29T23:34:28Z

@destimona Do you mean something happened the audio stream for the Firestorm viewers and the Second Life viewers at the exact same time?

I don't see any FMOD version in the system info you posted. It's possible that version is affected by the bug DJxScotpaul mentioned.

sl-service-account commented 3 years ago

DJxScotspaul commented at 2021-04-30T07:24:27Z, updated at 2021-04-30T07:24:53Z

perhaps id you read the ticket i put in months ago you would realise its happening to 1000s of ppl, its EVERY stream on sl not just my company

Its NOT possible and a waste of time me giving you my pc specs or to tell you how to reproduce it

Also the explaination from pantera should have guided you on the right track thats its fmod issue.

No wonder so many ppl are leaving secondlife for other platforms, because nothing gets fixed here !

sl-service-account commented 3 years ago

JIRAUSER336979 commented at 2021-04-30T14:17:30Z

At least once every hour while I'm DJing, my stream cuts out. Sometimes, more. It happens every day, on every sim, every time I broadcast. I am using Virtual DJ with 128kpbs stream from a reliable provider. It has happened with multiple providers on multiple sims. 

I've been DJing for about 2 years and this is a consistent issue. I've never had it not happen. 

sl-service-account commented 3 years ago

sandi Mexicola commented at 2021-05-02T09:46:13Z

I would like to report that I do not have this problem. :)

sl-service-account commented 3 years ago

Ansariel Hiller commented at 2021-05-02T14:34:26Z

Claiming

No wonder so many ppl are leaving secondlife for other platforms, because nothing gets fixed here ! while at the same time saying Its NOT possible and a waste of time me giving you my pc specs or to tell you how to reproduce it indicates the problem why "nothing gets fixed": Stating that "something does not work" but then not wanting to actively help in the process of getting it fixed - just saying...

sl-service-account commented 3 years ago

Destimona commented at 2021-05-02T17:11:45Z

Okay so if I could delete some of these comments I would. Ansariel, if you read the actual issue you'd see computer specs have nothing to do with it because it's not just one person, you'd have to compile data from thousands of people as far as computer specifications. This is why it was pointless. If you read the actual entire thread instead of picking apart one comment that'd be fantastic. 

sl-service-account commented 3 years ago

Destimona commented at 2021-05-02T17:12:32Z

Okay so if I could delete some of these comments I would. Ansariel, if you read the actual issue you'd see computer specs have nothing to do with it because it's not just one person, you'd have to compile data from thousands of people as far as computer specifications. This is why it was pointless. If you read the actual entire thread instead of picking apart one comment that'd be fantastic. 

sl-service-account commented 3 years ago

Ansariel Hiller commented at 2021-05-03T09:33:52Z

Guess what: I read the entire thread - so much for your wrong assumptions! And guess what else: Same as Sandi, I never experienced these issues myself. And maybe it doesn't happen for LL staff as well. So simply saying "it's happening everywhere for basically everyone" won't help anything at all - reason why Dan asked for details when and where it can be observed. If you are upset by having actively participate in the bugfixing process and provide specific details, you should be aware this usually ends in nothing getting fixed at all - simple as that!

Pro-tip: You could have added a log file of the viewer for the session it happened at least - that might contain details of why playback stopped.

sl-service-account commented 3 years ago

DJxScotspaul commented at 2021-05-03T09:49:13Z

Ansariel i did do all that its in the ticket link, im sure staff can read it and realise its affecting 1000s of ppl on sl, i have been working with this for 6 months

The problem is we cannot replicate an issue

However someone from Firestorm has said its an FMOD issue and will be fixed in the next update

 

sl-service-account commented 3 years ago

Dan Linden commented at 2021-05-03T23:35:33Z

I tested this for a couple hours on Friday after 3 PM and have not yet experienced either a pause nor a full stop of the music stream.

To anyone experiencing this bug, the next time this bug happens, take note of the time it happened and if it's a short pause or the music completely stops. Then please write that in a comment on this Jira issue and attach your SecondLife.log using More -> Attach files.

This page tells you where to find the logs folder: http://community.secondlife.com/t5/English-Knowledge-Base/How-to-report-a-bug/ta-p/733545#Section_.3

Thank you!

sl-service-account commented 3 years ago

Destimona commented at 2021-05-04T17:49:19Z

[10:44] ᴅᴊ ᴡᴏʟғ ᴄᴀɴswoʀᴛʜ ᴀᴢᴀʟᴇᴇ™ (canadianwolf913): Hello Pixi [10:44] Pιχι ƇЄƝƊƦЄƧ ƁƁ Scѳtspauℓ (destimona): Hey there How can I help :) [10:44] ᴅᴊ ᴡᴏʟғ ᴄᴀɴswoʀᴛʜ ᴀᴢᴀʟᴇᴇ™ (canadianwolf913): just wanted to let you know my music keeps dropping at Runaround Sues [10:45] Pιχι ƇЄƝƊƦЄƧ ƁƁ Scѳtspauℓ (destimona): send me a slurl and the time [10:46] ᴅᴊ ᴡᴏʟғ ᴄᴀɴswoʀᴛʜ ᴀᴢᴀʟᴇᴇ™ (canadianwolf913): Serena Montecito (84,89,31) [10:47] ᴅᴊ ᴡᴏʟғ ᴄᴀɴswoʀᴛʜ ᴀᴢᴀʟᴇᴇ™ (canadianwolf913): I am djing right now [10:47] ᴅᴊ ᴡᴏʟғ ᴄᴀɴswoʀᴛʜ ᴀᴢᴀʟᴇᴇ™ (canadianwolf913): I started at 10 am

sl-service-account commented 3 years ago

Destimona commented at 2021-05-04T18:00:28Z

for the last few weeks I've experienced my v-side stream dropping.

It has happened as I DJ at fluidity nightclub.􀀀 each and every time i dj there. My set was 10am-12 pm sl time.

I've since left that gig

 

Now the dropping is happening at my home sim, at my clubs.

􀀁 􀀂

its happening now at Fusion - 5:20 isn sl time pm

stream software is oblivious to the problem and keeps streaming, but the stream stops in sl, and everybody has to hit the music notes again.

This is becoming aggrevating.

This did not happen until the last month or so for me. Certainly not before converting the shoutcast version.

Thanks, Octavia Sorbet

sl-service-account commented 3 years ago

Destimona commented at 2021-05-04T18:00:50Z

My name is Montana Ballard I have had the stream drop several times

I lost my stream around 4 pm while I was DJing and had to reboot several times.

Prior to that I was unable to hear the previous DJ approximately around 3 pm and after I DJ was unable to hear the music from the next DJ.

http://maps.secondlife.com/secondlife/Black%20Stallion%20Country/197/173/26

 

 

sl-service-account commented 3 years ago

Destimona commented at 2021-05-04T18:07:06Z

Hot Steel (214,66,4043) 6 PM SLT Stream

http://maps.secondlife.com/secondlife/Zebria/179/44/2006 4PM Turbo Streams and 6 Pm VSide Streams 3PM onward http://maps.secondlife.com/secondlife/Black%20Stallion%20Country/197/173/26

 

sl-service-account commented 3 years ago

Destimona commented at 2021-05-04T18:07:28Z

My name is Montana Ballard I have had the stream drop several times

I lost my stream around 4 pm while I was DJing and had to reboot several times.

Prior to that I was unable to hear the previous DJ approximately around 3 pm and after I DJ was unable to hear the music from the next DJ.

http://maps.secondlife.com/secondlife/Black%20Stallion%20Country/197/173/26

 

sl-service-account commented 3 years ago

Destimona commented at 2021-05-04T20:37:47Z

My music turned off on it's on at 1 PM at http://maps.secondlife.com/secondlife/Zebria/190/39/2009

sl-service-account commented 3 years ago

Destimona commented at 2021-05-06T17:02:23Z

10 am  at http://maps.secondlife.com/secondlife/Zebria/192/50/2007 music turned off on its own 

sl-service-account commented 3 years ago

Destimona commented at 2021-05-06T17:11:49Z

DarkHeart (calypygyny) reported music dropping at 10 am http://maps.secondlife.com/secondlife/Zebria/192/50/2007

sl-service-account commented 3 years ago

Dan Linden commented at 2021-05-06T17:26:06Z

We need logs. Please see my last comment.

sl-service-account commented 3 years ago

Destimona commented at 2021-05-06T18:35:04Z, updated at 2021-05-06T19:05:48Z

so instead of the accurate answers I'm giving you from people all over, you want everyones computer logs as well?

sl-service-account commented 3 years ago

Dan Linden commented at 2021-05-07T01:25:37Z

The music stopped for me at 6:13PM at http://maps.secondlife.com/secondlife/Zebria/217/59/2007 on Second Life Release 6.4.18.558266. Attached "01. music cut out at 6-13 SecondLife.log" The logs from that moment say:

2021-05-07T01:13:20Z INFO # llaudio/llstreamingaudio_fmodstudio.cpp(223) LLStreamingAudio_FMODSTUDIO::stop : Stopping internet stream: http://vside-radio.com:8334
2021-05-07T01:13:20Z INFO # llaudio/llstreamingaudio_fmodstudio.cpp(109) LLStreamingAudio_FMODSTUDIO::start : Set internet stream to null
sl-service-account commented 3 years ago

DJxScotspaul commented at 2021-05-07T05:43:09Z

And this ladies and gentlemen is why things never get fixed on sl. You just can't fix stupidity. I have tried for 6 months to get  this fixed but when we come up against this type of thing then I give up. Good luck with this post your going to need it

sl-service-account commented 3 years ago

Ansariel Hiller commented at 2021-05-10T16:06:34Z

And this ladies and gentlemen is why things never get fixed on sl: Instead of actually providing logs, more comments and rants leading to nothing!

@Dan Linden

Since I assume you didn't press stop to stop the stream, this can only happen if FMOD either couldn't determine the stream state or there was an error with the stream itself (FMOD_OPENSTATE_ERROR). So the issue isn't directly SL-related, but could either be a problem within FMOD (highly unlikely since it seems to happen at particular times only), or some ISP / stream hosting provider / networking issue (network congestion, bandwith issue, stream server issue or something like that).

At this point, it might be a good idea to gather some data, what ISP and stream hosting provider affected users are using, but I can already imagine the "accurate response" you will get... ;)

sl-service-account commented 3 years ago

DJxScotspaul commented at 2021-05-10T16:43:44Z

we provided information but why oh why should i have to collect logs from every dj who has issues

BUT let me tell everyone i know with issues to come post here

and its not a rant its affecting 1000s of ppl

anyway you dont seem to grasp the issue so im out of this post

sl-service-account commented 3 years ago

AndreyK ProductEngine commented at 2021-05-10T17:27:22Z, updated at 2021-05-10T17:31:28Z

if FMOD either couldn't determine the stream state or there was an error with the stream itself (FMOD_OPENSTATE_ERROR) stop() is also called from inside start(), start() could have arrived from viewer (startInternetStreamWithAutoFade). Either some of the devs need to repro it personaly or we need better logging in and around fmod, plus a bit for parcel media. It's worth having a jira just to improve looging for these sections. And once we know what actually happens we might be able to fix that (if it isn't a network issue).

we provided information but why oh why should i have to collect logs from every dj who has issues A single log file that caught the issue would have been enough. Preferably more, but even a single one povides some data points.

sl-service-account commented 3 years ago

Ansariel Hiller commented at 2021-05-10T18:42:42Z, updated at 2021-05-10T18:47:31Z

@[~andreyk.productengine]

It cannot be called from within start(), because if it would, there would be at least some log output of one of these execution paths - which is not shown in Dan's log:

image-2021-05-10-20-42-11-740.png

Also, the implemention of LLStreamingAudio_FMODSTUDIO::update() appears to be wrong:

image-2021-05-10-20-30-20-961.png

image-2021-05-10-20-31-58-269.png

You don't even check the result of the FMOD::Sound::getOpenState() call, which may anything but FMOD_OK, yet you return state, which is FMOD_OPENSTATE_READY in that case. But either way, LLStreamingAudio_FMODSTUDIO::stop() is only called in case of FMOD_OPENSTATE_ERROR, which according to the FMOD API doc is:

image-2021-05-10-20-37-57-768.png

Unfortunately due to the incorrect implementation, you don't log the error condition of the getOpenState()-call, thus the log file doesn't contain useful information any further.

And of course it would be helpful to know what ISP / stream provider the affected users / DJs are using to find a common denominator among all the affected users - especially since the error is most likely caused by something going on on the network / stream host.

sl-service-account commented 3 years ago

AndreyK ProductEngine commented at 2021-05-21T15:15:05Z, updated at 2021-05-21T15:34:54Z

It cannot be called from within start(), because if it would, there would be at least some log output of one of these execution paths But isn't that exacly what dan's log shows? Or am I missing something?

Likely from start's string 99 'stop()', at 2021-05-07T01:13:20Z: LLStreamingAudio_FMODSTUDIO::stop : Stopping internet stream: http://vside-radio.com:8334 then string 109 LLStreamingAudio_FMODSTUDIO::start : Set internet stream to null

Dan points to this as being exacty the moment when audio stopped and this looks like something out of fmod caused this.

You don't even check the result of the FMOD::Sound::getOpenState() That does seem like it needs to be checked

sl-service-account commented 2 years ago

Dan Linden commented at 2021-12-01T19:54:05Z

There is a new Release Candidate viewer, "Maintenance Jenever", to try at https://releasenotes.secondlife.com/viewer/6.5.1.566306.html It has been updated to FMOD Studio 2.02.03, has some retry logic which will try to restart the music network stream if it drops out, and has additional logging to help us debug network stream dropouts. Please try the Maint J viewer and let us know if the music streams play more reliably.