[BUG-40875] Bring back weekly rolling maintenance for the grid

[sl-service-account]

How would you like the feature to work?

Back in the days, SL had regular rolling restarts every week. But then, it has began to become less frequent: every 2-3 weeks in average for main channel. Quite often, rolling restarts are even cancelled or not performed, despite they appear planned in the grid status page. They seem to occur only when it comes to new server updates deploy, which is a pity. However, without regular restarts, any region's performance and stability runs downhill. No 'but', no 'if', that's a fact. And simply a matter of time.

During the summer 2016, we have even been left with 6 weeks without a maintenance! As a result, the grid can become a true wreck for long days or weeks. The problem has actually gone even more acute since the prim limit was increased on regions. So please roll back to weekly restarts!

Why is this feature important to you? How would it benefit the community?

As an active member of the sailing community, I have witnessed numerous races or cruises that turned into a bloody mess due to the grid poor state and performance. As I'm writing this, around 50% of boat races I have followed had to be cancelled because nobody (or almost) survived them. The turning point has been the day since the region prim limit has been increased. Coincidence? I think not!

The problem is especially severe in Blake Sea and the surroundings, which is a hig traffic area where vehicle use is the main activity. However, boats, planes and even road vehicles need a grid in proper condition to be enjoyable. Instead of this, we more and more often meet region that are randomly misconnected to nearby ones and/ or running bad. As a result: lag, crashes, sim crossing / teleport failures, etc. As a further result: people getting fed up and quitting, not to mention that general feeling that SL is slowly dying...

It's simply not conceivable for residents to flood customer support with tickets for each region concerned and each time it happens (that is: more and more often). It's simply a waste of time and a joy killer. We are all in SL to have fun, not to spend our time filling in forms!

I strongly believe that a more frequent maintenance could solve that problem and avoid that defective regions are left to decay for too long. Therefore, I hereby request that the grid is restarted every week. If not the whole grid, at least the SL server main channel regions, where most of the transportation activity takes place.

Original Jira Fields

| Field | Value | | ------------- | ------------- | | Issue | BUG-40875 | | Summary | Bring back weekly rolling maintenance for the grid | | Type | New Feature Request | | Priority | Unset | | Status | Been Triaged | | Resolution | Triaged | | Reporter | indragray (indragray) | | Created at | 2016-11-11T20:02:29Z | | Updated at | 2016-11-28T06:35:09Z | ``` { 'Business Unit': ['Platform'], 'Date of First Response': '2016-11-11T16:17:13.255-0600', 'How would you like the feature to work?': 'Back in the days, SL had regular rolling restarts every week. But then, it has began to become less frequent: every 2-3 weeks in average for main channel. Quite often, rolling restarts are even cancelled or not performed, despite they appear planned in the grid status page. They seem to occur only when it comes to new server updates deploy, which is a pity.\r\nDuring the summer 2016, we have even been left with 6 weeks without a maintenance! \r\nAs a result, the grid can become a true wreck for long days or weeks. The problem has actually gone even more acute since the prim limit was increased on regions.\r\nSo please roll back to weekly restarts!', 'ReOpened Count': 0.0, 'Severity': 'Unset', 'Target Viewer Version': 'viewer-development', 'Why is this feature important to you? How would it benefit the community?': "As an active member of the sailing community, I have witnessed numerous races or cruises that turned into a bloody mess due to the grid poor state and performance. As I'm writing this, more than 50% of boat races had to be cancelled because nobody (or almost) survived them.\r\nThe turning point has been the day since the region prim limit has been increased. Coincidence? I think not!\r\n\r\nThe problem is especially severe in Blake Sea and the surroundings, which is a hig traffic area where vehicle use is the main activity. However, boats, planes and even road vehicles need a grid in proper condition to be enjoyable.\r\nInstead of this, we more and more often meet region that are randomly misconnected to nearby ones and/ or running bad. As a result: lag, crashes, sim crossing / teleport failures, etc. As a further result: people getting fed up and quitting, not to mention that general feeling that SL is slowly dying...\r\n\r\nIt's simply not conceivable for residents to flood customer support with tickets for each region concerned and each time it happens (that is: more and more often). It's simply a waste of time and a joy killer. We are all in SL to have fun, not to spend our time filling in forms!\r\n\r\nI strongly believe that a more frequent maintenance could solve that problem and avoid that defective regions are left to decay for too long. Therefore, I hereby request that the grid is restarted every week. \r\nIf not the whole grid, at least the SL server main channel regions, where most of the transportation activity takes place.", } ```

[sl-service-account]

Aethelwine commented at 2016-11-13T21:56:50Z

I am involved with several cruises amd sim crossings have been noticeably worse the last few weeks. Something that was working before, isn't working now and trying this seems likely to help.

[sl-service-account]

foneco Zuzu commented at 2016-11-13T21:56:58Z

Make sure you watch it as well, i add my vote and im watching.

[sl-service-account]

Tomomichan commented at 2016-11-13T22:06:52Z

Are these the new norms to be accepted? Cross a Sim and crash? This s killing Blake Sea and the movement thoroughfares for Boats and Planes. This is game ending. Get it right or lose it. Restarts are a basic necessity. Do It Linden Labs. Do it.

[sl-service-account]

Deirdre McMinnar commented at 2016-11-13T22:17:37Z

I'm not saying that you desperately need to fix this, Lindens... All I'm saying is that I have the money to put my kid through college in hand already.

[sl-service-account]

kittensusie Landar commented at 2016-11-13T22:47:37Z

I run weekly cruises with Leeward Cruising Club and some good friends organise flights for me with Skyward Flying Club, and I hate to see people deciding to give up a cruise or flight because sim crossings are so bad. This isn't just an inconvenience. Fewer people cruising or flying with us means lower earnings in tips for the DJ's at our post-cruise/flight parties, and for some of them, tips are their only source of income.

Please bring back the weekly restarts....

[sl-service-account]

Conte Alchemi commented at 2016-11-13T23:02:11Z

I absolutely agree with what everyone said here. Blake Sea area never performed so bad as lately, and a regular sims restart and maintenance is vital to keep the Blake Sea and the sailing and flying activities alive.

[sl-service-account]

atosuria.daviau commented at 2016-11-14T00:18:48Z

This also effects land owners of Coastal properties who rent Boat slips or airport Hangars or parking. If the sostomers go away from poor Sim crossing performance LL will have alot of abandoned land on there hand on the coasts. Without the value of Sailing or aviation they will sit empty.

[sl-service-account]

Serena Lexington commented at 2016-11-14T00:35:02Z

Each sim border crossing is a lottery. When you cross a sim (every 256m means pratically constantly when you fly), you never know if your going to make it on the other side. 90% of the crossing you loose your sight for around 5 seconds, your huds disapear or/and move on the screen to another location, blocking your view and their effect you have to recycle them in order to get them back to work, or, they can also simply detach, or die, become unusable but be undetachable (relogging being mandatory at this point... On almost all sim crossings, your avatar is bailed out your aircraft for 5 to ten seconds, folowing your aircraft from the edge of the sim you're crossing, letting you out of control of your aircraft, and often ending in the sea ruining your flight ONE MORE TIME ... So yes, COMPLETE rolling restart of ALL SERVERS every week would DEFINITELY be a minimum ...

[sl-service-account]

Gemma Vuckovic commented at 2016-11-14T01:30:57Z

I am Commodore of the Nantucket Yacht Club, and have been sailing in SL for over 9 years, way before the Blake Sea was formed. I fully agree and support with this JIRA, sailing not only in the Blake Sea, has always been difficult most of the time, but we are used to that. Sadly in recent years its become so bad that a lot of sailors have left SL. Not only Sailor, but Flyers too . I am fed up having to continually send in Tickets to complain about conditions. Yes, I am very happy with the responses from the Lindens, and sims are restarted when asked, but the problem is that its not long before another restart is needed. I get mixed messages too, one person tells me that sims are restarted every week, then next person tells they are not. Many many people love to come to sl for relaxation, and end up very angry. WE need a regular daily restart, not even weekly, the Blake Sea it extremely well used by sailors, cruisers and flyers.

[sl-service-account]

Brooke Leung commented at 2016-11-14T04:38:48Z

Please LL start doing rolling restarts again so that we can go back to enjoying the Blake. For those of us that rent property along the Blake to be able to enjoy sailing there, it has become a hardship and money wasted. Please do something and go back to weekly rolling restarts.

[sl-service-account]

Iteke commented at 2016-11-14T05:37:33Z

Traveling has become increasingly difficult, especially the last few weeks. You crash easier and will often have trouble crossing sim borders, not just in the Blake but also on the roads. I think it will be a good thing to have more restarts.

[sl-service-account]

lisa123 Sabra commented at 2016-11-14T07:20:37Z

What they said, please :)

[sl-service-account]

maury49 rau commented at 2016-11-14T09:58:22Z

i am a italian sailor, i hate to crash so i think LL must think to e daily restart, at least weekly. good sails to all

[sl-service-account]

Shakespeare Shamrock commented at 2016-11-14T15:44:18Z

i run search and rescue group and also enjoy the SLRR, i also love sailing warships and patrol boats, but even doing SAR is near impossible at times when you spin out control end up on bottom sea or flying outside of the sar vehicle, linden labs need fix this specially with extra prim count means more scripts etc etc

[sl-service-account]

JoyofRLC Acker commented at 2016-11-14T16:16:21Z, updated at 2016-11-14T16:17:30Z

I agree things have been very bad lately. My experience over the last eight years or so is that these things go in cycles, getting better, stabilising then, like snakes and ladders, it all falls apart again. Wash, rinse, repeat. Not sure of the root cause. Id say that this is not currently the worst its ever been, but close to it.

In brief I have 3 recommendations, which I'll explain below.

1. Do DEPLOYS no more than once a month.
2. Do RESTARTS ideally daily, or at least every other day

3. Make sure the minimap ACCURATELY shows

   • Sims CURRENTLY OFFLINE in RED (as is supposed to be the case currently I believe)
   • Sims ABOUT TO GO OFFLINE in YELLOW (new feature)

   So, yes I support the JIRA with slight modification.

   ....

   Please correct me if I'm wrong, but isn't the "weekly roll" actually a server Deploy (of updated software), as opposed to a simple restart? I have always felt that deploys are too frequent and that may be a cause of inadequately tested software. Also, the deploy is done in the "morning" for the majority of NA users (and, yes we are not the whole world, but certainly a significant block of users). Activities like sailing, powerboating, flying, jogging (?), etc etc etc are effectively shut down for 2 or 3 hours ... using rolling restart doesn't help as you need large numbers of contiguous sims online not just one. So from a use point of view too frequent deploys are an inconvenience. Presumably they are done at the time they are as systems staff probably need to be on hand in case (when) anything goes wrong.

   So I would advocate for Deploys being no more frequently than monthly at the very most, certainly not weekly.

Restarts on the other hand are another matter. Well run Private Estates do restarts daily or every other day. Restarts should be relatively quick, and trouble free (there is no change to the server software). I see no reason why LL could not do the same thing. And also LL could probably arrange to do it at whatever time typically has the least number of users on. I'm not sure, given that restarts are fairly quick, but also the need for contiguous sims for sailing etc, if the should be done on a rolling basis across areas like Blake Sea, or all at once. People more expert than I can figure that out.

A couple of related questions / suggestions. How confident are we that the minimap correctly shows which sims are off line? I think its very important that the map correctly shows sims that ARE offline (in red) AND ALSO sims that are ABOUT TO RESTART (say within 2 minutes), in Yellow, say. Then we can at least go around. They can give us the wobbly buzzing yellow pop up when a sim is about to restart, so colour the mini map as well!

Joy Acker Commodore, Fishers Island Yacht Club

[sl-service-account]

JoyofRLC Acker commented at 2016-11-14T16:20:53Z

A further thought. Perhaps LL should also be checking to see if there is any other cause for the current crashes / falling off vehicles. "Lag" is a pretty broad brush. Doing frequent restarts is probably the first step. But I would not bet the farm that it will fix the problem. At least we woujld then know to look further.

[sl-service-account]

NatashaFatale25 commented at 2016-11-14T17:08:21Z

I spent a lot of money, here, and was having a lot of fun, sailing. Then, inexplicably, the restarts stopped, and things got less enjoyable by the, day. Now, after having left, for all intent & purposes, a few months, ago, I have deleted my account. I do not intend to return. I really appreciated, what you Lindens did, for us...now, not so much. I'm sorry.

[sl-service-account]

NatashaFatale25 commented at 2016-11-14T17:09:29Z

P.S. Seems, as if, you are throwing all resources into Sansar, and have forgotten all about Second Life. Truly, sad.

[sl-service-account]

JessieMartinique commented at 2016-11-14T17:31:27Z

I'm sorry, but I am really getting fed, up! I left, to travel, awhile, and when I returned, a couple months, later, logged on, & tried to sail, again, I was greatly, dismayed! Seems the routine restarts have become, history?? Why, in thunder, would one stop DOING them!? Are you waving goodbye, and grinning all the way to the bank?? I used to have loads of fun, here! I spent a lot of money, getting stuff! I bought roughly 14 boats -expensive ones- and now I can no longer sail, them!? Come, ON! What are you, thinking?? Sansar sounds like it could be loads of fun, but to give us the finger on Second Life, at the same time, is NOT cool. I canceled my Premium membership, awhile back, precisely because, things were going wrong, and I was only getting lip service, for all my effort ot let you Lindens, know! Now, I am SERIOUSLY considering dumping all my stuff by deleting my, account! I AM FED UP! Now??? Not, so much. I AM, sorry! You, however, were the architects, of my attitude.

[sl-service-account]

Archer Mactavish commented at 2016-11-14T18:39:15Z

Please fix these issues!

[sl-service-account]

unclesamoriginal commented at 2016-11-14T18:58:16Z

I am a founder of the Gulls Wing Sailing Club. I am voting to support this JIRA, but want to note that the Gold Standard for 3d gaming grids should be a minimum of one restart a day. Most private sector businesses require computer shut downs at the end of a shift to save energy, but the effect is a fresh machine restarting for the next shift. Imagine, dear Lindens, if any service provider you hired did not provide you with fresh online availability of your data and processes! Your current practices will depopulate the public free areas of SL faster than you would fire your own service provider! The JIRA request for rolling weekly restarts is at best a minimalist remedy. If you implement it by itself, you are meeting at least a Bronze Standard. Perhaps you need to examine restart data from quality private estates to learn how managers in the trenches utilize restarts to maintain the quality of their operations.

[sl-service-account]

Ginnafer commented at 2016-11-14T19:00:03Z

If you think it is bad for boating, try flying an airplane. It is horrible. Why oh why did they ever stop rolling restarts on a weekly basis? Our Airline Black Sheep Commuter Air Service, has almost stopped all flights. We only fly after a restart is done.

[sl-service-account]

Herasdejavou commented at 2016-11-14T19:00:52Z

please back with the rolling restarts. in the moment big problems with sailing or fly. crashes and more

[sl-service-account]

Farris Fargis commented at 2016-11-14T19:23:45Z

I think LL is letting SL die a slow and painful death. I own a mainland region that I have to request restarts on. They're done only sporadically otherwise. Wake up, LL. You're losing it.

[sl-service-account]

RainbowSails commented at 2016-11-14T20:38:06Z

We desperately need a good running Blake Sea

[sl-service-account]

josynovecinque commented at 2016-11-14T20:54:45Z

ding ding ding, shame shame shame!

[sl-service-account]

Andrea Curtis commented at 2016-11-14T21:04:50Z

Blake Sea, and other ocean areas, are a huge resource for Second Life sailing, and proper maintenance of the grid is essential to the entire Second Life community. Definitely bring back routine rolling restarts.

[sl-service-account]

Hartman Jameson commented at 2016-11-14T21:15:25Z

Crashed on the Intercoastal around Pieve in an open sim .. took my boat and me to the bottom . cleared off my hair , feet and hands .. tried to go through again. Same thing ... just bad maintenance ..three times now in that area...

[sl-service-account]

Hartman Jameson commented at 2016-11-14T21:18:28Z

You need to do something about the virtual sailing ... It's one of the largest enjoyments in sl . Now everywhere is bugged out. I have a premium membership .. I am considering dropping it .

[sl-service-account]

Pehkhet commented at 2016-11-14T21:35:38Z

Can only second this. Please focus on smooth sim transitions and general stability... and until you can pull that off regular restarts seem to be the only way to ensure a somewhat decent SL experience.

[sl-service-account]

xeniaelisabeth commented at 2016-11-14T21:47:17Z

Sailing or flying through the Blake Sea and many other places) is becoming increasingly difficult. Sims refuse to appear or crash. This spoils he enjoyment of SL.

[sl-service-account]

Dirk Cerna commented at 2016-11-14T22:36:44Z

Sailing and Flying the Blake Sea has become treacherous please do something to fix it!!

[sl-service-account]

Cate Foulsbane commented at 2016-11-15T03:05:26Z

Sailing on the SL seas is the main reason I am in SL..and when sailing is difficult, my happiness diminishes. So many of us are affected by this..please please fix it!

[sl-service-account]

CleoSertorii commented at 2016-11-15T03:49:01Z

I got into boating in SL a couple of years ago because it was relaxing and gave me the opportunity to do something (at least virtually) that I am unable to do in RL. I enjoy it so much that I am willing spent more money to rent an island on one of the sailing estates connected to the Blake Sea sims. I also recently started flying helicopters just for some variety and have found out how fun that is as well. I don't know for sure if the problems in the Blake sims started with the increased prim allocations, but the timing is too close to ignore the possibility. In the mean time, I cannot sail or fly more than 2-3 sims at a time, before i get booted from my boat/helo, kicked into a sim corner, and then my viewer crashes. This was never an issue even as recently as a couple of months ago. I didn't spend all of these lindens on rent and boats/helicopters just to have to restart my viewer every 10 minutes. If rolling restarts will fix the problem with bad sim performance, then I am all for it. LL please listen to your residents :)

[sl-service-account]

Geneer Gynoid commented at 2016-11-15T03:56:13Z

I think the small percentage of inconvenience of a frequent restart far outweighs the remaining percentage of unstable sim activity. Thank You.

[sl-service-account]

Anu Daviau commented at 2016-11-15T04:53:46Z

Ladies and gentlemen at Linden Labs, there are people here who actually put serious money into Second Life, be it for their premium accounts, mainland, standalone sims or private estates. Can't we expect an appropriate maintenance of the servers?

[sl-service-account]

pleasureboi commented at 2016-11-15T09:35:34Z

It started getting far worse when the simulator code to add extra prims went in. Time to revert that and let us sail again.

[sl-service-account]

jazzydaze commented at 2016-11-15T11:09:03Z

Sailing is one of my favorite things about SL. Please fix, its rather messy right now!

[sl-service-account]

lale Delvalle commented at 2016-11-15T15:16:53Z

Sailing and or flying? It's about SL in general. I pay for this crap. Restarts weekly would be nice but what about maintenance. There's more to maintenance than just restarts. Restarts are good for region servers but do nothing for you. You your inventory and other things are not on sim servers and need maintenance like everything else. Everything works together and when one part is screwed it makes your and my SL life miserable. If you can't fix it, sell it to somebody who will invest and do what's right.

[sl-service-account]

retune commented at 2016-11-15T16:17:52Z

I started sailing about a month ago. I rarely can finish a course. It is so very hard to learn how to race in s/l as it is, but to be booted virtually EVERY time I sail is very discouraging. I am not a s/l or computer expert, but I think you are. Please do what is necessary to mitigate these events. I spent good money on my boat, and I LOVE this environment! Please help....thank you in advance.

[sl-service-account]

LeyanneMace commented at 2016-11-15T16:52:49Z

I'd really like to see something done about this, too

[sl-service-account]

indragray commented at 2016-11-15T17:21:09Z

Due to the popularity of this case, I have no choice but setting the priority from "undefined" to "major" ;-) If today, LL has performed a rolling maintenance on main channel regions, don't be mistaken: that operation was planned long ago and is not related to this case. BTW, just check the grid status page ( https://community.secondlife.com/t5/Status-Grid/bg-p/status-blog ) and search when the previous rolling (NOT deploy, just maintenance) was performed for main channel. Yes, you're reading it right: October 25th. 3 long weeks ago!

[sl-service-account]

Kyle Linden commented at 2016-11-15T19:25:47Z

Hi Indragray,

Thank you for filing this request. The team has reviewed it and we are investigating further.

[sl-service-account]

indragray commented at 2016-11-27T22:03:28Z, updated at 2016-11-28T06:35:10Z

Since friday 25th, the grid - and especially Blake Sea - has been crippled again. Random regions are showing poor performance (lag, rubberbanding effect, physics FPS collapsing, sim crossing fails, not connecting to nearby regions, etc. The whole range of bugs!). Once again, I went to a boat race and it was wasted: half of sailors (including me) simply abandoned at mid-race because they were simply fed up with crashing each time. As a results, I sent a ticket to customer support requesting a restart for the whole Blake Sea. To no avail. I was replied that I had to provide the name of a specific region for that :-( But when there are several defective ones? Do we all have to suspend our current activities to find and list them all? Which brings us back to what I said in the OP: "We are all in SL to have fun, not to spend our time filling in forms!"

Bottom line is: I am now under no illusion that grid performance begins to run downhill 9-10 days after a restart. At the moment, rolling restarts are processed only every 2 weeks for main channel. This is clearly insufficient! I opened this case asking for weekly restarts for a good reason. I am aware that this case has been triaged and melted under another case, but I insist on main channel regions are restarted more often, at least as a temporary fix, until the actual cause of these issues is found and solved.