[BUG-234214] Intermittent Scripting Issues after recent server rollouts...

[sl-service-account]

What just happened?

I have observed several scripts that were previously working failing to work after the last rollout. This appears to have effected the golf score system I have made as well as lucky chairs, and vendors scripted by other parties.

What were you doing when it happened?

Troubleshooting issues reported by those that have the score system I created. After which I learned of added issues that were similar to what I was seeing. Many of which appear to relate to events that are triggered within a timer or sensor failing to be triggered reliably sometimes resulting in the script becoming frozen without an error.

What were you expecting to happen instead?

I expected scripts that were running in a stable manner prior to the server rollouts in July of 2023 would continue to do so after such rollouts.

Other information

It seems the scripts are still running as using a listen event in a script that is frozen in this way still works and seems to un-stick the frozen script. This is re-triggered seemingly at random and I have as of yet been unable to figure out what is causing it. Some of these issues can still be seen on earlier versions of the score system that I made which were working perfectly fine prior to rollouts in early to mid July of 2023.

Addendum

After continued investigation from my own end I feel it necessary to make abundantly clear that this appears to effect not only the previously mentioned function calls. Calls, functions, and calculations using floating point numbers (including those used for location, distance, timing, and other measurements) appear to be intermittently effected. When a script is effected it may produce erroneous data from function calls, produce improper output from calculations, or in some cases the script may hang without crashing (that is to say that scripts impacted in this way stop processing anything within the current state or event handler without any crash or error, scripts that have hung are still reported to be active and running and have been observed to temporarily recover when provided outside stimuli for which said script is designed to respond and then re-hang after a short period of time).

Original Jira Fields

| Field | Value | | ------------- | ------------- | | Issue | BUG-234214 | | Summary | Intermittent Scripting Issues after recent server rollouts... | | Type | Bug | | Priority | Unset | | Status | Closed | | Resolution | Cannot Reproduce | | Reporter | Certified Lunasea (certified.lunasea) | | Created at | 2023-08-04T03:59:23Z | | Updated at | 2023-08-22T23:03:13Z | ``` { 'Build Id': 'unset', 'Business Unit': ['Platform'], 'Date of First Response': '2023-08-03T23:07:17.512-0500', "Is there anything you'd like to add?": 'It seems the scripts are still running as using a listen event in a script that is frozen in this way still works and seems to un-stick the frozen script. This is re-triggered seemingly at random and I have as of yet been unable to figure out what is causing it. Some of these issues can still be seen on earlier versions of the score system that I made which were working perfectly fine prior to rollouts in early to mid July of 2023.', 'ReOpened Count': 0.0, 'Severity': 'Unset', 'System': 'SL Simulator', 'Target Viewer Version': 'viewer-development', 'What just happened?': 'I have observed several scripts that were previously working failing to work after the last rollout. This appears to have effected the golf score system I have made as well as lucky chairs, and vendors scripted by other parties.', 'What were you doing when it happened?': 'Troubleshooting issues reported by those that have the score system I created. After which I learned of added issues that were similar to what I was seeing. Many of which appear to relate to events that are triggered within a timer or sensor failing to be triggered reliably sometimes resulting in the script becoming frozen without an error.', 'What were you expecting to happen instead?': 'I expected scripts that were running in a stable manner prior to the server rollouts in July of 2023 would continue to do so after such rollouts.', 'Where': 'Regions using the golf score system I created:\r\nhttp://maps.secondlife.com/secondlife/Sky%20City/218/32/51\r\nhttp://maps.secondlife.com/secondlife/Viper%20Isles%202/160/157/33\r\nRegion with multiple lucky letter chairs (one of which stopped working without errors):\r\nhttp://maps.secondlife.com/secondlife/HiVid/57/56/23', } ```

[sl-service-account]

Certified Lunasea commented at 2023-08-04T04:01:35Z, updated at 2023-08-04T04:03:50Z

Since beginning to attempt narrowing down this issue on my own it has come to my attention that server version 2023-07-31.581251 was rolled out to the main channel region that my wife and I rent on Tue 2023-08-01 07:14 PDT and further investigation reveals this has been rolled to much of the grid, but this server version does not have release notes. This has left me in the dark about what might have been altered and given the documentation about the server versioning currently in use this server version number appears to indicate insufficient testing time prior to being rolled out, though I hope that this is not the case and that release notes are going to be posted soon. I only mention this because I am not certain what was done there did not effect these previously stable scripts.

[sl-service-account]

JIRAUSER345862 commented at 2023-08-04T04:07:18Z

I have a golf course in SL and have been having intermittant timing and ball recognition issues using our previously solid scoring system. This has become suddenly much more of an issue after the recent roll outs.

Please review the roll out and perhaps pull it back for further testing. 

[sl-service-account]

Maestro Linden commented at 2023-08-04T18:27:46Z

Hi [~certified.lunasea], you can view the server release notes at https://releasenotes.secondlife.com/simulator/2023-07-31.581251.html

Could one of you please post some repro scripts so we can investigate the script failures?

[sl-service-account]

Certified Lunasea commented at 2023-08-04T19:04:31Z, updated at 2023-08-04T19:06:09Z

I have no access to the mentioned lucky letter chairs at http://maps.secondlife.com/secondlife/HiVid/57/56/23 as these are not my scripts. The score system I made is a released product comprised of several scripts and is able to be examined in a homestead or full region environment by going to http://maps.secondlife.com/secondlife/Sky%20City/218/32/51 or http://maps.secondlife.com/secondlife/Viper%20Isles%202/160/157/33 respectively. If a linden would like to contact me in world I would be happy to provide further information.

Unfortunately due to the intermittent nature of the issue I am unable to find a way to reliably reproduce the issue. That said the scripts were previously stable and working prior to rollouts beginning in early to mid July while I was offline for 10 days on vacation. Coming back to complaints of previously stable scripts breaking (including at least two releases of my own scoring system dating from 2023-04-11 and 2023-05-24 and that underwent extensive testing prior to their releases) is a pretty good indication that something else has changed.

Unfortunately it appears that there are missing links to that set of release notes on most of the pages at the releasenotes subdomain of secondlife.com so that information fails to appear on the main page of said subdomain or on several of the pages beyond that main page and it may be prudent for this to be corrected.

Again if more information is needed on how the scripts that I created work, then I would be more than willing to speak with a linden in-world to discuss the matter.

[sl-service-account]

Certified Lunasea commented at 2023-08-08T16:25:18Z

While I am unable to help when it comes to providing a reliable reproduction script due to the intermittent nature of the effects noticed or with any information regarding the frozen but running script in the lucky letter chair I mentioned in the report itself I can provide you with a list of locations you may obtain a free score card from my golf score system that is of a version that was previously stable prior to recent server rolls. The locations are as follows:

http://maps.secondlife.com/secondlife/Furrenburg/2/6/34 http://maps.secondlife.com/secondlife/DANAAN%20PARK/233/204/27 http://maps.secondlife.com/secondlife/Sunset%20Palm%20Resort/110/123/762 http://maps.secondlife.com/secondlife/Willow%20Wilds/28/113/22

Locations currently using up to date versions of the score cards created after 2023-07-16 (which may still be effected though to a lesser degree) are as follows:

http://maps.secondlife.com/secondlife/Sky%20City/213/36/51 http://maps.secondlife.com/secondlife/Viper%20Isles%202/162/151/33

The issues are intermittent when present, and with newer versions I had to modify the code in attempts to work around these issues. Should you need further information I am willing to provide what information I can about the effected scripts that I wrote privately.

[sl-service-account]

Maestro Linden commented at 2023-08-11T22:54:17Z

I visited one of the SLURLs with lucky chairs, and gave them a look. They have a lot going on, and also have the ability to communicate with the public internet, so we'll need the script creator (who appears to be [~Ben.Vortex] ) to add debug messaging to identify exactly where it's getting stalled.

Specifically, need to know things such as

1. Whether the script is failing to trigger an expected event
   • e.g. no link_message() when a message was sent to it, or if the dataserver() event failing to trigger or returning incorrect data when reading a notecard
   • In this case we'd want to know which event was getting messed up, and hopefully have an idea of the data that was expected
2. Whether a certain LSL function is returning the wrong value with specific parameters, causing the script to get stuck
   • In this case we'd want to know which parameters were fed to the function, and what the context was, such as looking up data about an avatar who was in the sim
3. Whether scripts are somehow losing permissions previously granted by avatars

4. Whether scripts are crashing unexpectedly

   If you can identify exactly where a script is breaking, feel free to send the script or parent object privately.

[sl-service-account]

scullyuk commented at 2023-08-12T02:27:26Z

Hello,

I am not aware of any issue with my lucky chairs, I am not sure where this information has come from.

Thanks, Ben Vortex

[sl-service-account]

Certified Lunasea commented at 2023-08-12T08:02:39Z

At the time of the incident I was asked to take a look by an employee at HiVid as one of the 5 chairs appeared to be running but had stalled operations. After examining it myself and observing attempts by multiple avatars, including myself, to sit upon the chair (with and without the first name beginning with the displayed letter) we found that the chair was entirely unresponsive as it should unseat users that have a first name with a letter not matching that displayed, and those matching it should be given the item displayed. Checking for running scripts using the script info capabilities in the latest version of the Firestorm viewer revealed the scripts in the chair were all running, but still the chair refused to update data or react to users at all. Unfortunately the employee at HiVid was unable to reset the object and due to the way most chairs operate we determined the issue must be that the script had frozen and the employee at HiVid reported that no crash was observed and that the chair had been working earlier while he was there. The employee eventually covered it with caution tape textured prims to dissuade users from using it.

 

I cannot discount the possibility of a crash to that particular chair, but that would need to be something that Ben would provide added information for. To be sure Lucky Letter chairs tend to be relatively simple scripts (even if they use dataserver triggering functions, link messages, or even HTTP requests) and with 4 others running identical code sitting on the same parcel and region it is odd that only one was effected and that all scripts reported as running. That all said more than the Lucky Chairs from Ben have been effected. In my own scripts I was able to narrow things down to the timer or sensor events which had been working perfectly since 2023-04-11 and 2023-05-24 when I last made releases before these issues began. This is code I could have provided to you for your examination and which suffered from the same issues unexpectedly.

 

I am curious to find out if anyone checked for such issues in the operation of the score systems present on the six regions for which I provided SLURL locations, if not then it may already be too late to do so at said locations as I sent out an update to customers as of yesterday that was an attempt to try to mitigate the issues that those customers began to see in early July. I am not 100% certain that my adjustments will be successful, but the adjustments were relatively minor. I still find it very odd that no issues were observed with the previous versions of said code prior to the rolls in early July and would very much like to know what happened to cause such issues.

 

I would like to make it known that I have been online nearly every day since filing this Jira as I have been attempting to work around this issue after it effected my own previously stable scripts. I am rather disappointed that even after offering to provide added information I have not been contacted to provide such. Ben even contacted me to ask for information as he was unaware of what was going on, which I would assume was due to being on vacation, as the HiVid employee told me that was the case. I am rather easy to reach, and while I am not keen on publicly posting my code for a product currently released to users in-world, the product is indeed available for review and if need be I would be happy to open it up for Linden Lab examination for the sole purpose of attempting to either solve the server issues observed, or at least find out what has changed to cause the difference in behaviors so that wiki documentation can be updated and/or residents informed about such changes. With that said, if you should need additional information please do reach out and let me know.

[sl-service-account]

Maestro Linden commented at 2023-08-14T17:27:03Z, updated at 2023-08-14T17:28:49Z

Hi [~certified.lunasea], the only LSL-related changes between the current 2023-07-31.581251 simulator build and the previous one, 2023-06-09.580543, are just these points from the release notes:

• Animesh can now be identified through llGetPrimitiveParams
• New LSL functions llLinksetDataDeleteFound and llLinksetDataCountFound
• Fix for BUG-233830: DATA_PAYINFO information would not always expire from the cache

Your content couldn't have been using the newly-introduced llLinksetData..() functions in the old simulator version, so that leaves just the animesh feature (available in the PRIM_SCULPT_FLAG_ANIMESH bitfield in https://wiki.secondlife.com/wiki/LlGetPrimitiveParams ), and the DATA_PAYINFO change (related to https://wiki.secondlife.com/wiki/LlRequestAgentData ). It seems unlikely that the sculpt flag is related to the chairs' operation, but it's possible that they're using llRequestAgentData.

At this point we need reproduction steps for the issue, demonstrating an issue with the LSL API - for example, if there's a pattern in which llRequestAgentData returns the wrong data for the agent or returns no data at all, that would be something we could investigate and make a demonstration script that illustrates the bug. Likewise, if you believe the issue is with the sensor or timer functions, we could use a test script there.

We haven't had any other recent reports of LSL suddenly behaving differently, so I unfortunately can't offer any direction that would help to narrow down the issue.

[sl-service-account]

Lucia Nightfire commented at 2023-08-15T16:43:13Z

The only weirdness I experienced recently was that, even though request keys were returned and correct, llRequestSimulatorData() and DATA_SIM_STATUS would return empty data for several dozen different regions between 2023/08/12 01:32:27 and 2023/08/12 01:37:17 slt.

[sl-service-account]

Certified Lunasea commented at 2023-08-16T13:19:33Z

Though the issue does appear to effect specifically sensor and timer functions/events I cannot rule out the possibility of it being something to do with the llMessageLinked function call or the link_message event handler, either as each script of my own that I notice this occurring with uses those functions to pass information back and forth between scripts. And since you mentioned the PRIM_SCULPT_FLAG_ANIMESH feature within llGetPrimitiveParams I have to ask if that change may have also effected llSetPrimitiveParams function calls as well since the items I have scripted and have noticed having these issues are also using these function calls (though they do not use PRIM_SCULPT_FLAG_ANIMESH in any way). Since the scripts appear to hang, rather than crash (no error is produced when they stop functioning and they are reported to still be running by the system), I am at a loss to find the issue.

 

With regard to my score system for golf courses across Second Life I did notice that outside input through a listen event appears to temporarily un-hang the script as it will still process that input. Unfortunately it doesn't cause the script to resume running properly as sensor and timer events still appear to be effected within them. This issue caused players to have penalty strokes added to their game even when it previously would not have, and still should not have, occurred. While I have managed to somehow mitigate those issues for the most part with recent updates to the scripts (for which 4 iterations were required) I am still seeing the issue occur from time to time in other items. I have kept pretty meticulous records for the score system (I have no less than 31 revisions of such in my inventory ranging from v1.25b to the current v3.12). At the time of the issues versions 3.07 and 3.08 were released (and copies of these are still in my inventory). These are the versions for which I got complaints of golf ball detection issues and which had been in customers hands since April 29 and June 15 respectively. All I can really do for you here is narrow down the time frame from when things were working and when changes appeared to occur. I can also tell you that to help in mitigating sensor issues in my score system the flags used to find the ball had to be changed from SCRIPTED (which was used from v1.25b through v3.08 and was working without fail) to SCRIPTED|ACTIVE|PASSIVE (which was changed as of v3.12) and such had to be done without any alterations having been made to the balls used by players. In addition I can tell you that the sensor call is not used in other previously stable scripted objects that I have noticed having issues with timers to display data or alter primitive parameters.

 

If the code for the Second Life servers from Linden Lab did nothing to alter these functions in any way that could potentially cause such odd failures then is it perhaps possible that the underlying server software maintained by Amazon for the AWS server stacks could have impacted what is running on top of it (including the Second Life servers)? If so did they potentially make alterations to said servers around the same time?

 

Regardless of what the changes were within that time frame it does not change the fact that these scripted objects that were working prior to these server versions began to experience issues after they were rolled out. I would say that appears to indicate that there was some sort of change, and that such change appears to have impacted those previously stable scripts. Again I would be happy to provide examples of the effected scripts to a Linden employee for the sole purpose of assisting with this issue, but I would need to know to whom I should send them as I am not able post them publicly given they are part of currently released products. That said, sending them to a Linden employee specifically for the purpose of troubleshooting this issue could help determine what might have changed from the time these scripts were working to the time in which they started having issues.

[sl-service-account]

JIRAUSER345977 commented at 2023-08-17T04:43:00Z

My golf ball made by Fa Nyak is detecting water on mainland where there is no water anywhere nearby.

<248.75160, 11.08598, 53.45507>

Water Height is 20.000. 

This is a mainland region.

http://maps.secondlife.com/secondlife/Mujoul/249/11/55

https://gyazo.com/562af19501cb5790081d12f5c19b3a30

 

[sl-service-account]

Maestro Linden commented at 2023-08-17T15:25:16Z, updated at 2023-08-17T15:26:00Z

Hi [~zildjian.clifton] you should file a separate Jira if you feel like LSL is returning incorrect water height information. That said, I cannot reproduce that issue. Iran a quick check of this function (which is the only way a script would know water height) in Mujoul: https://wiki.secondlife.com/wiki/LlWater

My test script is:

default
{
    touch_start(integer detected)
    {
        vector offset = ZERO_VECTOR;
        llOwnerSay("Water height is " + (string)llWater(offset) + " at "
            + llGetRegionName() + (string)llGetPos());
    }
}

The output I see is: > Water height is 20.000000 at Mujoul<249.00000, 11.00000, 56.01242> - so it's detecting the a 20m water height in the region, which matches what the viewer shows.

Setting this jira back to "Needs More Info" for the original issue about potentially broken lucky chairs.

[sl-service-account]

Certified Lunasea commented at 2023-08-18T08:01:18Z

Maestro Linden, please do allow me to quote the original post for the betterment of understanding of the issue at hand:

"I have observed several scripts that were previously working failing to work after the last rollout. This appears to have effected the golf score system I have made *as well as** {*}lucky chairs, and vendors scripted by other parties.{\}"

 

Perhaps now you can understand my disappointment at your disregarding my offers to provide scripts I have written to a Linden Employee for the sole purpose of figuring out what is causing these intermittent issues. Please understand that I have been willing to help, and my efforts have come at the cost of my own time and are for the purpose of attempting to ensure that this issue is understood and properly handled.

 

The issue here *IS NOT* about lucky chairs or any specific script for that matter. IT *IS* about the Intermittent scripting issues we are experiencing {}after the recent rollouts{}. These issues seem to effect sensors, timers, and distance calculations to varying degrees and seemingly at random (hence the statement that they are INTERMITTENT), unfortunately being that these issues do appear to be intermittent it is difficult for us to trace and pin down.

 

As for needing more information, I have repeatedly offered to provide scripts to a Linden Employee solely for the purpose of troubleshooting this issue. These are scripts I have written that have experienced these same issues, but they are not able to be posted publicly as they are scripts which are in currently released items. Items and scripts which worked perfectly fine as they were written until early July when these rollouts began.

 

The issue Zildjian Clifton is seeing is indeed similar to what we are seeing in relation to sensor and distance calculation issues where such intermittently fail to work as they should. In Zildjian's case this ball was at 53m altitude at the given coordinates. llWater being passed those coordinates should return a water height of 20m. The way these ball objects have detected if they are under water has worked for several years. The method used has been to compare its current z position with the water height and should its z position be less than water height the ball reports that it is underwater. This is well known behavior in each ball used for playing golf within Second Life in the Freestyle, Crowley Corp, and the yet to be released True Golf club systems. Since the method by which this is done is known, and considered to be stable, we know that if the ball reports being below the water at that location then either its math is no less than 33m off from what it should be, or llWater has returned an erroneous value. I think this marks a significant, albeit intermittent problem, and means that this previously known stable method for detecting if an object is underwater may now need to be double-checked where it did not require such previously (and for which doing so will increase script loads if it is possible at all given at least one of those creators is reportedly unable to re-enter Second Life).

 

While we look into that you might also wish to look into and address why it is that one of the recently released server versions was rolled out to the entire grid without the possibility of being adequately tested given that the versioning scheme used by Linden Lab for the region servers indicated that it rolled out to the entire grid within about 24 hours after final compilation of that code. I test my code rigorously, as do many scripters, coders, and programmers both in and out of Second Life, we would expect the same of Linden Lab.

[sl-service-account]

Signal Linden commented at 2023-08-22T23:03:14Z

Thanks for the report, Certified. Unfortunately, we haven't been able to reproduce the problems you are describing and have not seen any elevation in reports of similar problems from other users.

An example reproduction scenario, including an example script attached to a jira ticket is tremendously useful, even if the script needs to run for some period of time to exhibit the issue. If you only have proprietary scripts that you do not wish to disclose I suggest you attach a test case script designed to reproduce the issue based on the specific behavior in the script experiencing issues.