secondlife / jira-archive

2 stars 0 forks source link

[BUG-772] Simulator refusing to rez objects after 10 hour timeframe #15396

Open sl-service-account opened 11 years ago

sl-service-account commented 11 years ago

Steps to Reproduce

Nothing

Actual Behavior

Since the past 6 days I receive complaints from residents that they cannot rez on their island and receive the following error message:

Unable to create item that has caused problems on this region

Issues did start after Rolling Restart of previous week.

When I go to the island myself and attempt to rez an Object from my Inventory I receive the same message.

Region restart does solve the issue but only temporary for a period of 10 hours. After this period any rez attempt results in the above mentioned error. Also an avatar that logs in directly to the region is naked without any attachments.

Pathfinding is not enabled on this region so this cannot be the cause of the issue either. I did attempt to turn off scripts. I did check for temp on rez objects. I did turn off physics. I did look for unusual behaviour but nothing shows up.

I do know that this is a widespread issue at the moment and many people have this error at the moment on their island. http://community.secondlife.com/t5/Land/12-09-Second-Life-Unable-to-create-item-that-has-caused-problems/qaq-p/1728059

Your support department did attempt to solve the issue by placing the island on a different server. Placing the island on LeTigre server code, placing the island back on normal server code but that did not make any difference. I do not see anything related to performance of scripts that can have an impact on this. Everything was working correct until 6 days ago.

Three support tickets have been openened the past 6 days. Several contacts with support did not make any difference so far. Support did go to the islands to look but was not able to do anything except taking note of the bug themself.

Four members of your support staff were unable to do anything to solve this.

All of them could not do anything.

I have two islands affected by this for almost a week now.

Expected Behavior

Normal performance of your software.

Other information

I would like to see this solved fast as I cannot keep restarting simulators 3 times per day in order to keep them functional.

Thank you

UPDATE DECEMBER 6 2012

Two residents arrive on the island Dreamworld Rhino and commence to terraform on their island. Until the arrival of the residents the island performance has always been ok. The day the residents arrive who are both using the Phoenix Viewer memory consumption goes from 32 MB to 550 MB in a short time cycle. When I am on the island and watch while the residents terraform I see how the physical memory climbs with 100 MB or more in 30 minutes time. In two hours the residents bring the island from 30 MB to 250 MB and make it impossible to rez any objects in the region.

I did find this on the internet: http://pastebin.com/X6r6y5eW

Over a 3 week time cycle no improvement has been made. It is not possible to restart an island every two hours while residents terraform their island until it becomes defunct. When I terraform myself on the island the memory consumption does not move at all, I did test this. This appears to be triggered by the Phoenix third party viewer. This would explain why this issue is not more spread out.

Links

Duplicates

Original Jira Fields | Field | Value | | ------------- | ------------- | | Issue | BUG-772 | | Summary | Simulator refusing to rez objects after 10 hour timeframe | | Type | Bug | | Priority | Unset | | Status | Accepted | | Resolution | Released | | Reporter | Count Burks (count.burks) | | Created at | 2012-11-14T06:04:32Z | | Updated at | 2023-07-18T18:06:16Z | ``` { 'Business Unit': ['Platform'], 'Date of First Response': '2012-11-14T10:35:59.691-0600', "Is there anything you'd like to add?": 'I would like to see this solved fast as I cannot keep restarting simulators 3 times per day in order to keep them functional.\r\n\r\nThank you', 'System': 'SL Simulator', 'Target Viewer Version': 'viewer-development', 'What just happened?': 'Since the past 6 days I receive complaints from residents that they cannot rez on their island and receive the following error message:\r\n\r\nUnable to create item that has caused problems on this region\r\n\r\nIssues did start after Rolling Restart of previous week.\r\n\r\nWhen I go to the island myself and attempt to rez an Object from my Inventory I receive the same message.\r\n\r\nRegion restart does solve the issue but only temporary for a period of 10 hours. After this period any rez attempt results in the above mentioned error. Also an avatar that logs in directly to the region is naked without any attachments.\r\n\r\nPathfinding is not enabled on this region so this cannot be the cause of the issue either.\r\nI did attempt to turn off scripts. I did check for temp on rez objects. I did turn off physics. I did look for unusual behaviour but nothing shows up.\r\n\r\nI do know that this is a widespread issue at the moment and many people have this error at the moment on their island. http://community.secondlife.com/t5/Land/12-09-Second-Life-Unable-to-create-item-that-has-caused-problems/qaq-p/1728059\r\n\r\n\r\nYour support department did attempt to solve the issue by placing the island on a different server. Placing the island on LeTigre server code, placing the island back on normal server code but that did not make any difference. I do not see anything related to performance of scripts that can have an impact on this. Everything was working correct until 6 days ago.\r\n\r\nThree support tickets have been openened the past 6 days. Several contacts with support did not make any difference so far. Support did go to the islands to look but was not able to do anything except taking note of the bug themself.\r\n\r\nFour members of your support staff were unable to do anything to solve this.\r\n\r\nAll of them could not do anything.\r\n\r\nI have two islands affected by this for almost a week now. ', 'What were you doing when it happened?': 'Nothing', 'What were you expecting to happen instead?': 'Normal performance of your software.', 'Where': 'http://slurl.com/secondlife/Dreamworld%20Jaguar/134/142/30\r\n\r\nhttp://slurl.com/secondlife/Dreamworld%20Swan/137/155/45', } ```
sl-service-account commented 11 years ago

Maestro Linden commented at 2012-11-14T16:36:00Z

Hi Count, thanks for the report. I visited you region this morning, and didn't have any trouble rezzing objects. But I do see that as recent as Nov 13 20:44 (2 restarts back), objects were failing to rez in the sim. The reason for the rez failures was that the physics engine was nearly out of memory.

You can view the physics memory usage in the statistics bar, under 'Simulator' -> 'Physics Details' -> 'Memory Allocated'. Right now your sim is at 67MB, which is normal. When object rezzing began to fail, your sim was probably approaching the homestead physics memory limit of 256MB; the sim disallows rezzing when it gets within 90% of this limit (~230MB).

This physics memory bloat might be a bug, or it could be expected behavior, depending on what's occurring in the sim. Could you please monitor the 'Memory Allocated' stat in your region, and note which particular activity in the sim leads to a sharp increase in that stat? With details like "Rez object Z at spot Y, then observe that it permanently increases physics memory usage by xMB", we can attempt to reproduce this problem.

sl-service-account commented 11 years ago

Count Burks commented at 2012-11-16T12:30:36Z

November 16 3.14 AM SLT

The region Dreamworld Jaguar is at 360 MB Memory Allocated under Physics Details. Rezzing does fail again. I did check the region 6 to 8 times yesterday and did see how the memory consumption slowly increases during the day.

The region Dreamworld Swan also refuses to rez any objects and is at 128 MB Memory Allocated. The memory consumption in this region did increase from 36 MB to 128 MB in an 18 hours timeframe. Still the memory consumption is not near 230 MB in this region and the issue is exactly the same as on the Dreamworld Jaguar island.

Both regions appear to be suffering from something that drains memory from them or a leak of some sort. Rezzing certain objects in the region does not lead to higher spikes in consumption. The increase goes in a gradual way during the day.

sl-service-account commented 11 years ago

Maestro Linden commented at 2012-11-19T19:50:53Z, updated at 2012-11-20T17:50:28Z

Hi Count, both regions have low physics memory usage right now (50MB and 40MB), and object rezzing works fine.

Can you please let us know when the regions return to the bad state, so we can investigate the memory usage live? EDIT: it turns out that one needs a special debug simulator build to have Havok memory profiling (which can't go to Agni), so disregard this question. At this point, I'm waiting to see if your region has physics memory bloat on aditi; if so, we can move the Aditi copy to the debug build for analysis.

Also, that case where rezzing failed in Dreamworld Swan when it only used 128MB of physics memory in use could be caused by another problem in the sim - can you point to some specific time when that failure occurred?

sl-service-account commented 11 years ago

Maestro Linden commented at 2012-11-19T22:38:14Z

I launched a copy of Dreamworld Jaguar to see if the memory bloat happens when there are no visitors.

sl-service-account commented 11 years ago

Maestro Linden commented at 2012-11-20T22:46:32Z

It seems that navmesh rebaking by itself is insufficient to make 'Dreamworld Jaguar' permanently exhaust its physics memory limit; I tried rebaking the region a few times on Aditi, and the closest I got was ~200MB allocated for a short period of time (which dropped precipitously after about ~1 minute).

sl-service-account commented 11 years ago

Maestro Linden commented at 2012-11-26T22:31:09Z

Importing for investigation. I can't permanently reach the physics memory limit of a homestead on demand, but it's pretty easy to temporarily hit the 256MB limit.

sl-service-account commented 11 years ago

Caleb Linden commented at 2013-01-02T19:25:21Z

In other words, since the Physics Memory Allocated is higher than the limit, it's not possible to rez from the inventory.

sl-service-account commented 11 years ago

TJ Linden commented at 2013-01-26T20:00:36Z

I found similar symptoms on Thassainsel, a full region, and it's physics memory usage was showing at 900+ MB - I'm guessing that fulls have a limit of 1024 MB, this would hit right about the 90% usage mark as well. If this would be useful for testing, it only took about 8 hours from restart to hit this cap (probably less, it was 8 hrs when I tested it).

sl-service-account commented 11 years ago

Whirly Fizzle commented at 2013-02-03T20:44:02Z

Heya Maestro,

Possible clue as to what can cause a full region to reach its mem allocated limit is on BUG-1564 - the useage of the terraforming tools S&S Land Leveler and the EdenScape Terraform Brush.

sl-service-account commented 9 years ago

Vix Linden commented at 2014-11-07T23:55:16Z, updated at 2014-11-08T00:08:01Z

Ironjaw hit with the bug today, stats recorded at 3:42PM SLT: Virtual Memory at 282.1 MB, FPS 2 , Time Dilation .08