search

secondlife / jira-archive

2 stars 0 forks source link

[BUG-10627] llHTTPRequest appears to return NULL_KEY even if throttle is not reached #935

Open sl-service-account opened 8 years ago

sl-service-account commented 8 years ago

Steps to Reproduce

Objects that check backends via HTTP occasionally started failing after Tuesdays restarts, most notably Skill Gaming Objects.

Actual Behavior

llHTTPRequest seems to return a NULL_KEY when called, even if you have not reached the HTTP throttle. This is extremely intermittent, but once it occurs is a serious problem, because no response ever gets returned to the http_response event.

Expected Behavior

A valid key to be returned to llHTTPRequest and the http_response event to receive a response.

Other information

Sample code which could repro the issue (The repro we used passed <HTTP_METHOD, "POST", HTTP_MIMETYPE, "application/x-www-form-urlencoded"> list into llHTTPRequest. The below example does not do this however).

key qHTTP;

SendHTTP() { qHTTP = llHTTPRequest("http://www.google.com", [], ""); llRegionSayTo(llGetOwner(), 0, "DEBUG [" + (string)llGetTimestamp() + "]: llHTTPRequst called. Key = " + (string)qHTTP); }

default { touch_start(integer n) { SendHTTP(); }

http_response(key request_id, integer status, list metadata, string body) { llRegionSayTo(llGetOwner(), 0, "DEBUG [" + (string)llGetTimestamp() + "]: HTTP/" + (string)status + " Received from key " + (string)request_id); llSetTimerEvent(30.0); }

timer() { llSetTimerEvent(0.0); SendHTTP(); } }

Attachments

Original Jira Fields | Field | Value | | ------------- | ------------- | | Issue | BUG-10627 | | Summary | llHTTPRequest appears to return NULL_KEY even if throttle is not reached | | Type | Bug | | Priority | Unset | | Status | Accepted | | Resolution | Accepted | | Created at | 2015-11-06T03:44:12Z | | Updated at | 2017-05-08T23:34:57Z | ``` { 'Business Unit': ['Platform'], 'Date of First Response': '2015-11-06T08:02:16.993-0600', "Is there anything you'd like to add?": 'Sample code which could repro the issue (The repro we used passed [HTTP_METHOD, "POST", HTTP_MIMETYPE, "application/x-www-form-urlencoded"] list into llHTTPRequest. The below example does not do this however).\r\n\r\nkey qHTTP;\r\n\r\nSendHTTP()\r\n{\r\n qHTTP = llHTTPRequest("http://www.google.com", [], "");\r\n llRegionSayTo(llGetOwner(), 0, "DEBUG [" + (string)llGetTimestamp() + "]: llHTTPRequst called. Key = " + (string)qHTTP);\r\n}\r\n\r\ndefault\r\n{\r\n touch_start(integer n)\r\n {\r\n SendHTTP();\r\n }\r\n \r\n http_response(key request_id, integer status, list metadata, string body)\r\n {\r\n llRegionSayTo(llGetOwner(), 0, "DEBUG [" + (string)llGetTimestamp() + "]: HTTP/" + (string)status + " Received from key " + (string)request_id);\r\n llSetTimerEvent(30.0);\r\n }\r\n \r\n timer()\r\n {\r\n llSetTimerEvent(0.0);\r\n SendHTTP();\r\n }\r\n}', 'ReOpened Count': 0.0, 'Severity': 'Unset', 'System': 'SL Simulator', 'Target Viewer Version': 'viewer-development', 'What just happened?': 'llHTTPRequest seems to return a NULL_KEY when called, even if you have not reached the HTTP throttle. This is extremely intermittent, but once it occurs is a serious problem, because no response ever gets returned to the http_response event.', 'What were you doing when it happened?': 'Objects that check backends via HTTP occasionally started failing after Tuesdays restarts, most notably Skill Gaming Objects.', 'What were you expecting to happen instead?': 'A valid key to be returned to llHTTPRequest and the http_response event to receive a response.', 'Where': "This is REALLY affecting skill gaming objects at the moment, but I've heard it happening to breedables, plantpets, etc as well.", } ```
sl-service-account commented 8 years ago

vexacion commented at 2015-11-06T04:06:50Z

This shows the repro code reproducing the error inside of a skill gaming region on my skill gaming account.

sl-service-account commented 8 years ago

vexacion commented at 2015-11-06T04:24:05Z

I should also note - I have this reproducer code running in several regions. Right now, it has yet to fail in a single non-skill gaming region, and fails within 15 minutes in every single skill gaming region. However, I don't know if the problem is a skill gaming region, or if the problem is the amount of HTTP activity in the simulator itself. Skill gaming regions are going to have considerably more overall HTTP activity than a typical non-skill gaming region.

sl-service-account commented 8 years ago

Jasonballer SLSGO commented at 2015-11-06T14:02:17Z

this is really impacting operations of skill gaming regions please prioritize it as high/critical

sl-service-account commented 8 years ago

Caleb Linden commented at 2015-11-07T00:03:32Z

Hi vexacion,

Could you report us any additional failures starting 11/6 4pm PST, please? Click on Info Provided if you do.

Thanks!

sl-service-account commented 8 years ago

DragonfyreGames SLSGO commented at 2015-11-07T00:35:41Z

Caleb - I tested in one of my regions again, and it seems to be working allright, however, another operator passed me a screenshot which I'll attach within the last 30 minutes of having issues. Another operator also got a support response concerning this issue where she was told "The latest update changed the HTTP Throttles for llHTTPRequest". Is this true?

I'll attach the screenshot provided by the other operator to this issue. Notice at the end it finally gets the key.

sl-service-account commented 8 years ago

vexacion commented at 2015-11-07T00:37:36Z

This is the image provided by the other operator as mentioned in the above comment (Sorry, was logged on as my other avatar when I posted that update).

sl-service-account commented 8 years ago

Whirly Fizzle commented at 2015-11-07T02:17:50Z

The Sculpt Studio creator & group are also seeing many complaints about Sculpt Studio giving the following new error message on multiple regions: “Too many HTTP requests too fast. (Set HTTP_VERBOSE_THROTTLE, FALSE in the parameters to hide these messages.)”

sl-service-account commented 8 years ago

Caleb Linden commented at 2015-11-07T02:25:32Z

Hi Whirly,

Could someone from the Sculpt Studio group file a jira as well? Reporting with the current instance of HTTP request throttles as of 11/6.

Thanks, Caleb

sl-service-account commented 8 years ago

TheBlack Box commented at 2015-11-07T02:45:37Z

Were the LSL-http throttles changed ? Or is this an artifact of a server bug ?

If the throttles have changed:

sl-service-account commented 8 years ago

Whirly Fizzle commented at 2015-11-07T03:54:15Z, updated at 2015-11-07T04:18:43Z

@Caleb Do you want a seperate JIRA issue for Sculpt Studio repros? TheBlack Box who commented above me is the Sculpt Studio creator.

I just reproduced it so I'll give details here unless you say otherwise.

Repro 2

sl-service-account commented 8 years ago

vexacion commented at 2015-11-07T04:22:10Z

I'm going to set a high priority for this ticket based on Whirly's repro in a non-skill gaming region.

sl-service-account commented 8 years ago

TheBlack Box commented at 2015-11-07T04:37:08Z, updated at 2015-11-07T05:05:10Z

Just a heads-up: the problem with SS might be totally unrelated ... it seems we have a server-outage on our side, coinciding with the Sim-update

But if there was a change to lsl-http-request throttling, I still need to know about that and adjust accordingly.

Edit: Our server seems to be fine again. But the throttling is throwing a fit still. “Too many HTTP requests too fast." As far as I can tell, this problem got introduced with the sim update. And I don't know if I am supposed to adjust to a new throttling-regime, or if I am supposed to wait for LL to fix it.

sl-service-account commented 8 years ago

Whirly Fizzle commented at 2015-11-07T04:49:50Z

Haha now you tell me :D

I'm not sure it's unrelated tbh because the filers script also reproduced the problem at the same time as the sculpt studio slices returned the "Too many HTTP requests too fast" script error. I stopped testing anyway :)

sl-service-account commented 8 years ago

TheBlack Box commented at 2015-11-07T05:10:40Z

Thanks Whirly :)

I think you are right. I updated my comment above, after I came to the same conclusion.

sl-service-account commented 8 years ago

vexacion commented at 2015-11-07T05:14:21Z

We're in the same boat, TheBlack Box, not sure if we need to modify tons of objects we've written with entirely new throttle mechanisms or wait for LL to fix it. The llHTTPRequest throttle change has only been mentioned from a support response to another operator who had forwarded it onto me. The exact words he said was: "Recent changes in our server code have introduced throttles on HTTP connections. In most cases, you can correct the problem by removing the objects temporarily to allow the throttle to reset, then add them again."

To me, this seems like an absolutely poor solution. It also seems like this throttle is insanely low, as my repro script only does one request every 30 seconds and still fails.

sl-service-account commented 8 years ago

TheBlack Box commented at 2015-11-07T05:42:33Z

Good to know, vexacion.

Yeah. I am currently assuming that LL has to fix this. If there was supposed to be a change in the throttling, I would have expected some kind of information being made available before this rolled out.

sl-service-account commented 8 years ago

Jacob Forsythe commented at 2015-11-07T06:30:10Z, updated at 2015-11-07T06:31:31Z

Hey guys. Sculpt Studio owner and user here as well. 32 Slices do not give the error, but 64 slices do. Also. the Sculpt MAP does not get made from the mat to the server. Doesn't seem to make it from LL servers to the Sculpt Studio server. Notecard and OBJ files get produced. Not sure if they are complete though.

Would be nice to have this fixed ASAP. A lot of people count on this to work! Need to remove other things that really effect performance rather than things like this! Bewebs and Butts are a couple that I can think of!

Let me know if you need any more info.

Second Life 3.8.6 (305981) Oct 13 2015 17:30:25 (Second Life Release) sim10231.agni.lindenlab.com (216.82.49.153:13019) Second Life Server 15.10.23.306566

sl-service-account commented 8 years ago

ChinRey commented at 2015-11-07T13:39:52Z

I assume this change is caused by the recent incidents with sims collapsing because of too much net time. If so, would it be possible to introduce a http request throttle per sim rather than or in addition to the one per object? A single object doing a http request every second isn't going to do any harm and it's often necessary. The problem only arises when there are several such objects int he same sim at the same time.

sl-service-account commented 8 years ago

TheBlack Box commented at 2015-11-09T02:33:21Z

Note:

this problem does not seem to exist on Second Life RC Magnum 15.10.29.306991

only on Second Life Server 15.10.23.306566

Fix or additional information regarding a new throttling regime would be appreciated ...

sl-service-account commented 8 years ago

Caleb Linden commented at 2015-11-09T17:24:44Z

Hey guys,

Thanks for the report. We are checking it out.

sl-service-account commented 8 years ago

Caleb Linden commented at 2015-11-14T01:11:22Z

We bumped up the throttle could you check again please?

sl-service-account commented 8 years ago

Whirly Fizzle commented at 2015-11-14T20:04:37Z

As far as I can tell, that seems to have done the trick for Sculpt Studio. I'm now able to have 2 sculpt studio mats rezzed out, each working with 256 slices and I can generate both 256 slice sculpties at the same time without hitting the throttle. Previously that would have hit the throttle on each attempt. Just using a single mat on the region was causing the throttle to trigger before.

If I attempt to generate 3 256 slice sculpts at exactly the same time, this will hit the throttle, but tbh that's pushing it anyway & I was being deliberately evil ;)

sl-service-account commented 8 years ago

Lucia Nightfire commented at 2015-11-14T20:13:43Z

https://www.youtube.com/watch?v=7edeOEuXdMU

Forgive the link, but it is relevant to the statement above.

sl-service-account commented 8 years ago

vexacion commented at 2015-11-16T02:35:02Z

So far things seem to have settled down and are now working again. Before closure of this issue can we get clarification of any throttle changes that will be or already were made? The patch notes that started this issue stated nothing but support seemed to be aware that there were throttle changes and those changes were never communicated to anyone publicly as far as I can tell. Obviously changing something like the HTTP throttle is a really major change for scripters who use HTTP in their scripts, so we would like to know what the plan is moving forward if there is a plan to change this throttle, and ask that any proposed changes be announced in advance. Thank you.

sl-service-account commented 8 years ago

Damien Dusk commented at 2015-11-18T19:23:45Z

Agreed, it would be very helpful to get some sort of official statement on the limits with the new throttle. I do not understand why Linden Labs has been so secretive about this and did not warn people it was going to happen.

sl-service-account commented 8 years ago

Lyle Maeterlinck commented at 2015-11-21T05:39:26Z

I'm currently experiencing this issue on sims with large numbers of items that request URLs and report them to an external system on sim restart. If they don't get a success message, they request a new URL and try again in 5 seconds, that would be one HTTP request per object every 5 seconds, for a count of a few hundred items. None of the requests succeed until I deactivate a certain number of them, and then the requests start going through. I'm not getting any script errors, though. Are HTTP requests now being throttled on a per owner basis or based on some other criteria?

sl-service-account commented 8 years ago

Whirly Fizzle commented at 2015-11-21T21:45:04Z

^ Lyle filed a bug for his problem at BUG-10748.

sl-service-account commented 8 years ago

Lyle Maeterlinck commented at 2015-11-24T22:34:12Z

Whirly, The problem I note above I believe is a separate problem from the one I filed at 10748. What I mention above appears to be a throttle based on object owner, and 10748 is an intermittent html error page that appears to be from the squid cache.

sl-service-account commented 8 years ago

Lyle Maeterlinck commented at 2015-11-25T01:10:02Z, updated at 2015-11-25T01:11:01Z

Looks like the llHTTPRequest wiki was just updated yesterday with more information on throttles: http://wiki.secondlife.com/wiki/LlHTTPRequest. There are now per-owner throttles of 1000 requests per 20 seconds. This is breaking a lot of stuff for me. It would have been nice to have a warning on this, or at the very least, some kind of notification at the time of the change, so we aren't feeling around in the dark and scrambling to figure out what's going on. Can we get some kind of notification process for breaking API updates?

sl-service-account commented 8 years ago

vexacion commented at 2015-11-25T01:58:03Z

Thanks for the heads up on the wiki update, Lyle. I have noticed I had a few HTTP issues going on today but nothing like originally. Questions to LL: Is this HTTP Throttle per object owner based grid-wide or is it a per-region basis? Is there a way to tell how far into the throttle we are? Most object owners have no idea how HTTP works in the objects, nor have any idea on whether or not they are reaching the throttle yet the onus is put on them and one stray object could break a bunch of legitimate objects. llHTTPRequests are now sim based resources (on an object owner basis) and as such should have some kind of interface to check how many are currently being counted toward the throttle, such as llRequestURL does with the number of open URLs. This way an object owner can check for an object that is flooding the requests.

sl-service-account commented 8 years ago

DragonfyreGames SLSGO commented at 2015-11-25T19:22:17Z

This is not functioning as intended on the wiki. The throttle appears to trigger after only 100-150 requests and after about 12-15 seconds on average. This occurred in multiple regions.

I have provided a clear way to reproduce below.

  1. Download "Master" Script here: http://pastebin.com/AmYeGq9D
  2. Download "Slave" Script here: http://pastebin.com/cAEDcVM1
  3. Create a sphere and put the Master script in the sphere.
  4. Create a box and put the Slave script in the box.
  5. Duplicate the box 11 additional times, creating 12 boxes total.
  6. Each of the slave boxes will send one llHTTPRequest per second. In order to hit the 1,000 throttle per user limitation shown in the wiki, we would have to create considerably more than 12 boxes total. However, 12 boxes creates the failure, and only 100-150 requests are sent until the throttle is reached.

(Please clean up when you are done, as this is hitting a test.php file on our server - thank you :) ).

Output from our testing:

[10:49:22] JIRA BUG-10627 (Master): BLOCKED AFTER 18.112610 seconds (150 calls) [10:50:29] JIRA BUG-10627 (Master): BLOCKED AFTER 23.490050 seconds (124 calls) [10:51:40] JIRA BUG-10627 (Master): BLOCKED AFTER 19.733030 seconds (159 calls) [10:52:46] JIRA BUG-10627 (Master): BLOCKED AFTER 14.156590 seconds (118 calls) [10:53:25] JIRA BUG-10627 (Master): BLOCKED AFTER 13.533450 seconds (109 calls) [10:54:30] JIRA BUG-10627 (Master): BLOCKED AFTER 13.511030 seconds (107 calls) [10:55:37] JIRA BUG-10627 (Master): BLOCKED AFTER 15.667400 seconds (130 calls) [10:56:43] JIRA BUG-10627 (Master): BLOCKED AFTER 14.244420 seconds (118 calls) [10:57:47] JIRA BUG-10627 (Master): BLOCKED AFTER 12.755840 seconds (105 calls) [10:58:52] JIRA BUG-10627 (Master): BLOCKED AFTER 13.200000 seconds (107 calls)

sl-service-account commented 8 years ago

Caleb Linden commented at 2015-11-25T20:51:34Z, updated at 2015-11-25T21:21:28Z

Thanks for the detailed steps, Dragonfyre. I check it out

Follow up question: Which regions did you see the throttle kicking in at the rate you listed above?

sl-service-account commented 8 years ago

Lyle Maeterlinck commented at 2015-11-25T21:18:42Z

Hi DragonfyreGames, Thanks for setting up this test. I just tried it on two of our sims, and I got a different result on two different sims.

At Liquid Labs (Estate / Homestead), running RC Magnum channel, 15.11.13.307797, I was able to get up to 42 boxes and I didn't get throttled. At Liquid East, running "Second Life Server" main channel, 15.11.13.307797, I got your same result, throttled at 12 boxes. Then re-tested on Liquid Labs, ok at 12 boxes.

Liquid East is one of the regions where I've been experiencing the problem with the throttle. I didn't think there was any way I was hitting 1000 http requests in 20 seconds. I would say there might be an absolute max of 200-300, and only right after the region starts and everything on the sim checks in with the server with their new sim URLs.

sl-service-account commented 8 years ago

Caleb Linden commented at 2015-11-25T21:22:24Z, updated at 2015-11-25T21:55:02Z

Hi Lyle,

Checking out Liquid East myself momentarily

I have a repro set up and going to import it. Thanks for all the information!

sl-service-account commented 8 years ago

vexacion commented at 2015-11-26T02:38:43Z

RA CORE and FP Enigma both run "Second Life Server" 15.11.13.307797 as well IIRC. I planned to try a premium sandbox later tonight and once that's done I'll post those results.

sl-service-account commented 8 years ago

vexacion commented at 2015-12-03T08:40:17Z

Caleb – Region "Cleopatra" is having serious HTTP Throttle issues. I can barely get a single request to go through wrecking havoc on a lot of objects I have in that region on my skill gaming avatar. Please investigate this region as well when checking into the throttles. It's still a problem. While I have quite a few objects in this region that make HTTP calls, it should still be well below the 1000 limit. I had called support concerning this specific region a few days ago and they said "The virtual memory use in this region is through the roof that would break a lot of HTTP calls, have the EM reboot it and it should fix itself". They did reboot it, but it's still broken. I do not have this issue in other regions nearly to the extent I do in Cleopatra.