How would you like the feature to work?

Edit: a better suggestion for the key name was given in the comments, and I have struck-through the original and added it here.

llGetAgentList accepts a list of Options as a parameter: https://wiki.secondlife.com/wiki/LlGetAgentList

I suggest the following option parameter: TO_LINKSETDATA, string keyNamePrefix

When llGetAgentList returns a list of every avatar matching the scope flags, this option causes the list to populate into Linkset Data storage, rather than into script memory. The keys used are keyNamePrefix followed by the ~~list index number~~ UUID found. The return value of the function is a list containing a single integer success or failure code from the linkset data write operations rather than the entire list of UUIDs.

For example: list avatarsInRegion = llGetAgentList(AGENT_LIST_REGION, [TO_LINKSETDATA, "avi" ]);

Would return a list [LINKSETDATA_OK] if the operation was successful, and the LSD would contain the following key value pairs in the form of avi, where is the uuid that would have appeared in the returned list, had it not been redirected to LSD. The Value can be set to 1, or any constant, and I suggest that if Accepted request https://jira.secondlife.com/browse/BUG-232312 gets implemented, then relevant data from the supplied filter options could be used in place of the value (AGENT_LIST_REGION_ENTRY_START_TIME or AGENT_LIST_REGION_ENTRY_END_TIME)

Why is this feature important to you? How would it benefit the community?

One of the giant yellow warning boxes on the llGetAgentList page is the reason here. There's far too many ways this essential feature can break scripts in ways outside of our control. For example, any script helping with the running of an event (gives out items, makes annoucements, keeps class attendance, enters a raffle, notifies staff of arrivals) would reasonable need to know who to interact with, but the amount of memory consumed is unknown until the list is fetched.

With the introduction of new Event regions, this is especially a growing issue. 100 avatars maximum (which matched the maximum number in an Estate) was already enough UUIDs that script-makers needed to be careful about how much byte-code and other data they used. With the introduction of event regions, I can see reasons to want the 100-avatar cap raised, but the problem will only worsen.

On the other hand, we've been granted 128KiB of Linkset Data recently, and that data is available without forced delay and does not count against script memory. It's like a giant database for storing large dynamic lists...just the thing this function returns.

At present, we could fetch the list, hope our script didn't crash, and then send the entire contents of the list back up to the simulator as LinksetDataWrite calls; this increases our script size, sim memory use, and number of commands the script has to issue to the sim... Or, with this feature, we call the function and indicate at call-time that we ultimately plan to put it into LSD anyways, and it gets done without all the extra resources used.

Generally in all LSL functions where a script needs to fetch data from the simulator without knowing whether it'll cause a crash-like llGetAgentList-it would be incredible to be able to divert the data into LinksetData storage. No chance of crash, data is where we were going to put it, everyone wins.

Edit:

See also comments below, where I've written a bit more about the fuller sense of why I've suggested this nomenclature and a sort of overall direction that should help scripters know what to expect from LSL functions while also hopefully benefitting simulators with fewer intermediate processing steps and less heap management.

Links

[ ] https://github.com/secondlife/jira-archive/issues/12323

Original Jira Fields

| Field | Value | | ------------- | ------------- | | Issue | BUG-234317 | | Summary | Add LSD option to llGetAgentList | | Type | New Feature Request | | Priority | Unset | | Status | Closed | | Resolution | Unactionable | | Created at | 2023-08-25T03:50:32Z | | Updated at | 2023-08-30T18:16:21Z | ``` { 'Build Id': 'unset', 'Business Unit': ['Platform'], 'Date of First Response': '2023-08-25T01:22:36.716-0500', 'How would you like the feature to work?': 'llGetAgentList accepts a list of Options as a parameter: https://wiki.secondlife.com/wiki/LlGetAgentList\r\n\r\nI suggest the following option parameter:\r\nTO_LINKSETDATA, string keyNamePrefix\r\n\r\nWhen llGetAgentList returns a list of every avatar matching the scope flags, this option causes the list to populate into Linkset Data storage, rather than into script memory. The keys used are keyNamePrefix followed by the list index number. The return value is a list containing an integer success or failure code from the linkset data write operations.\r\n\r\nFor example:\r\nlist avatarsInRegion = llGetAgentList(AGENT_LIST_REGION, [TO_LINKSETDATA, "avi" ]);\r\n\r\nWould return a list [LINKSETDATA_OK] if the operation was successful, and the LSD would contain the following key value pairs in the form of avi# where # is an incrementing integer for each result found.\r\n', 'ReOpened Count': 0.0, 'Severity': 'Unset', 'Target Viewer Version': 'viewer-development', 'Why is this feature important to you? How would it benefit the community?': "One of the giant yellow warning boxes on the llGetAgentList page is the reason here. There's far too many ways this feature--which is extremely useful--can break scripts outside of our control. For example, an script helping with the running of an event--something that gives out items, makes annoucements, etc--would reasonable need to know who to interact with, but the amount of memory consumed is unknown until the list is fetched.\r\n\r\nWith the introduction of new Event regions, this is *especially* true. 100 avatars maximum was already enough UUIDs that script-makers needed to be careful about how much byte-code and other data they used; events, where this function is extra-useful--now make it much worse.\r\n\r\nOn the other hand, we've been granted 128KiB of Linkset Data recently, and that data is instantly available and does not subtract from script memory. In cases where a script needs to fetch data from the simulator without knowing whether it'll cause a crash--like llGetAgentList--it would be incredible to be able to divert the data into a separate storage that can handle 1000 UUIDs. No chance of crash, everyone wins.\r\n", } ```

JIRAUSER341305 commented at 2023-08-25T06:22:37Z

There's 101 ways someone might want to handle such a thing. I'd advocate for a strftime style %-deliminated format string, with %u being the uuid, %n being such an incrementing number, etc.

I'd think even better alternative, would be to trigger the sensor event (or similar, perhaps pass 0 as the number detected in this case — something I'd presume the regular sensor event will never see), with the usual llDetected functions having access to the entire list (held internally — and presumably more efficiently). … and you can populate LSD yourself however you want.

Or, even better still, an iterator approach. If we could find out when someone joined (as a timestamp), then you have a command that lets you ask, "who was the next person to join after X". You give it 0 to start a request, and just feed in the timestamp from the last person to get the next one until you run out of people. (I raised a similar idea back in BUG-232768, too, though I think timestamp would be even easier, and something we often burn script memory for already — ie. who's joined since I last looked.)

There's also the proposal to let returned values not actually occupy script memory at all — if it's issues can be sorted out. (Or my variation on the idea BUG-233819.)

There are plenty of ways to avoid crashing due to returning a long list, other than just moving the list into LSD instead (and the tsunami of linkset_data events that'll cause).

JIRAUSER332931 commented at 2023-08-25T15:17:08Z, updated at 2023-08-25T15:24:40Z

Oh yes, I need to make a separate request at some point about one of the points you raise here Bleuhazenfurfle–I wish there was a push-style notifier event for when an avatar arrives in a region! Leaving a region admittedly is a much harder ask, but at the least it ought to be a knowable and useful event when someone arrives and it would save tons of simulator slice time (I think) to be able to have the sim push "here's the one or several people who just arrived" as an event rather than requiring us to poll a list (and diff the list) on a regular time loop. And it would take advantage of some of that free hidden data in the way events do with llDetectedX(index) functions.

For this request, though, I'm mostly trying to pursuade our LL masters to consider a class of functionality in the TO_destination sort sense, for cases where we do need the data placed somewhere for us. In other words, the handful of functions (such as in the request you mentioned) that fetch an unknown length of foreign data and risk heap-colliding us, it would be very nice to start thinking in terms of allowing the scripter to decide where the data is sent.

As a practical example, I suggest TO_LINKSETDATA as a great new resource, because I see quite a lot of potential ways that scripters might want to send quite a lot of things there (more space, free up script memory, use of regex filters), and all cases where we can cut out the middle-ware parsing code and just put the data where it was meant to go should be lighter on region memory use and on garbage collection.

But while we're brainstorming a feature, let's put yours in, too! Why not allow an option for [TO_TOKENSTRING, string separator]. Returns a list with a single element: the list of UUIDs, with the separator inserted.

And, my ultimate goal: [TO_DATASERVER]. Returns a list with the UUID of the query ID generated by the event; dataserver event is raised when the data is ready to be accessed, and thn, over in the dataserver event, we'd have a wealth of ways to async get the length of the list and process entries etc.

Basically, I want [TO_destination]as a concept, and the new LSD is my first target because I can think of an instant application that solves a widespread problem that will be getting worse as event regions get better...but let's plan for more!

Peter Stindberg commented at 2023-08-25T20:31:15Z

llGetAgentList is inherently broken. Wrangling with it for 3 years now taught me a lot, and I think I harnessed it by now and beat it into shape (at least my scripts run on regions with 100 agents), but that unused second parameter shows that there were plans for it down the road, which simply never happened.

I'm not gonna lie: Its broken-ness is a competitive advantage for advanced scripters. But any simplification or way of filtering what kind of results you get, would be helpful.

Lucia Nightfire commented at 2023-08-26T01:49:51Z

If LL won't give us saving notecards to LSD https://jira.secondlife.com/browse/BUG-232848 then they probably won't give us saving agent lists to LSD.

On a serious note, though, if/when https://jira.secondlife.com/browse/BUG-232312 is implemented then we'll have modularity with llGetAgentList().

JIRAUSER332931 commented at 2023-08-26T05:02:33Z

Oooh, I like, @Lucia! I managed not to find that searching up the Jira and wish I had; it's compatible and possible cooperative! (Could have the usual list of flags and parameters, including all the filters and flags there, plus gaining some way to send the results to storage so we don't have to choose between polling often versus holding a list)

JIRAUSER341305 commented at 2023-08-26T06:36:49Z, updated at 2023-08-26T06:42:53Z

The trouble is, most of that is just kicking the can down the road a little.

TL;DR; Add LIST_AGENTS_SINCE and LIST_AGENTS_COUNT options to allow iteration/batching, and the mandatory loop we'll all still have anyhow pretty much supersedes all your TO_xxx options. (Edit: Also, Lucia's one (which I'd forgotten about also) is basically an only slightly flawed version of mine (though I like my names better). I mention below the problem with using llGetUnixTime as the timestamp… It'd do the job, but…)

To further expand my original counter-suggestion a little; a last joined timestamp (and associated LIST_AGENTS_SINCE option) really seems the best option to me, along with a LIST_AGENTS_COUNT option to set a batch size (iteration is just a batch size of 1, after all), as long as the timestamps resolution is high enough to differentiate individual people joining; either by using a simple counter instead of an actual time (for example, people joined since last sim restart), or by building a custom timestamp (like using the low 23 bits of llGetUnixTime, and an 8-bit counter of "people who joined this second", aught to be enough — though it'll wrap every 97 days, which is a bit of a pain). You could get away with llGetUnixTime as the timestamp, but at the cost of having to keep around the list of people who joined during the last second, so we can exclude any who joined immediately after the next time — an increase in both complexity and storage, which isn't required by just using a better timestamp. Yet another alternative; on the assumption that a sim has on average less than one join per second over a week (presumably even the most busy shops and info hubs have quiet periods), you may even be able to just use llGetUnixTime itself, until a second avatar joins in the same second, at which point you simply increment the last time for each new arrival until there's a pause allowing it to fall behind the current time again — though I'd argue that gives you the worst of both worlds.

The important aspect of this, is that once the script is able to loop over the list one at a time (or at least in small batches) rather than trying to hold it all in memory at the same time, it can do with it what it wants. The script can implement your TO_LINKSETDATA suggestion itself with a simple two-line loop. (TO_DATASERVER and TO_TOKENSTRING also become completely redundant.) This is after all basically what we do with the list anyhow once we've got it, we loop over it — and more often than not what we want to save is only part of that data anyhow, so even with TO_LINKSETDATA, we almost certainly still have to loop over it to do what we need to do. and with the simplistic form your suggestion takes, quite probably then immediately throw the whole lot out again ready for the next time. My suggestion of a format string (which I'll explain a bit more below, too) was aimed to help reduce that by making those entries actually useful as a persistent store of the information your script needs (or, alternatively, keeping it to the absolute bare minimum required).

Looking at how I interpret your TO_DATASTORE and TO_TOKENSTRING suggestions; Assuming LL take the most simplest naive approach, and store the array of UUID's with an event (as per your TO_DATASTORE), they can store them in binary form (16 bytes each) rather than the list of keys we need (102 bytes each), for a memory saving of 8.6kB for 100 avatars — if I haven't messed up my math, it saves 6 times as much memory as that internal array would use in it's entirety. With your TO_TOKENSTRING suggestion, that saves the event nonsense, and brings that 102 down to 74 bytes each, but still a massive waste over the 16 each that LL could manage by keeping the list internal. On top of that they may even be able to share that one array should two scripts happen to request the list in the same frame. (Either way, these methods still need to snapshot the current list of agents present though, because the script likely won't get to do all it's processing within a single frame, and it'll be LSD all over again if anyone joins or leaves in the meantime.)

Overall, my suggestion of iteration (rather than simply providing a whole stack of alternative storage options), is about not only letting LL manage the "list" in whatever way is most efficient for them, but as I described it, {}completely removes the snapshotting issue entirely{}. If a join/part happens after the cursor, you'll pick it up as you go, if it happens before the cursor, you simply pick them up next time, the important part is that your cursor remains stable even when the list is changing around it. And all that's required for that, is an ordered cursor of sufficient resolution — such as the timestamp options I mentioned, or the last key processed (combined with sorting) would be for LSD. This is the single immeasurably valuable benefit of using an iteration approach over anything else that's been suggested, and the reason why databases offer cursors. That timestamp is essentially filling the same role as a database cursor (we can actually use any other ordered unique value too, like username or uuid, but timestamp makes the most sense).

Back to my variation on your TO_LINKSETDATA suggestion; I'd just want a pattern string as I suggested, basically a key/value template (either one string with a separator, or two strings) producing records fully populated with the information I specifically want, picking from at least the basic stuff; uuid (%u) , username (%n), display name (%d), position (%p), rotation (%r), perhaps velocity (%v), time joined (%t), that index number of the item in this call (%n) that you mentioned, etc. I'd probably have it return the count of avatars, but also the count of new entries created (as opposed to replaced entries), by putting the uuid in the key instead of a count (as my modification to your suggestion allows), you can now determine immediately whether anyone joined or left without even having to look at the data. Likely most importantly though, my template version also lets you pick which field should be the key field, and how much (if any) additional data is stored along with it.

Combine this with an option to only add new LSD records (not modify any existing ones), and being that you can also shove static data into the template too, we now get the full package in one pass; we have people who left (via the old favourite llGetAgentSize, and removing the unneeded entry), as well as people who just joined (via having the newest generation count/timestamp), without having to keep a separate list of who was here last time. (I do imagine myself most often implementing the equivalent of TO_LINKSETDATA anyhow, via that loop, but using much more customised task-specific records, and with the ability to edit, not just add/replace records, that no TO_xxx option could ever cover — hence why I suspect even with my templating suggestion, TO_LINKSETDATA will still rather often result in a continuous populate-loop-wipe cycle.)

Lastly, expanding a little more on TO_DATASTORE. I think that one is basically one I've suggested already myself (I just put it in terms of TO_SENSOR instead, because I like the idea of reusing many of those llDetected functions we already have). I'm hoping you didn't mean a datastore event per agent, because that would be horrible — for the exact same reason several people have apparently sworn off ever putting a linkset_data event in their scripts. And that is a problem also suffered by your TO_LINKSETDATA suggestion, unless LL also add a LINKSETDATA_MULTIUPDATE action, or something, and lets not get started on how it repeats the evils of the past (llGetAgentList for example) by trying to cram a list of deleted/updated keys into a string parameter. WHY LL WHY?!?!?

JIRAUSER332931 commented at 2023-08-26T18:10:14Z

Interesting points, all!

I do agree that in the general sense, Lucia's separate (and happily, accepted) request is the best implementation for most needs with getting timely updated lists; better overall for diffing lists and working out who has come in since when. I think where I'm aiming with the suggestion of TO_LINKSETDATA is the other cases where storage and later manipulation of that list would be wanted. Basically, it's an options (plural) array, so I certainly wouldn't want TO_LINKSETDATA in place of/instead of Lucia's suggestion, as that suggestion is the more broadly useful and has the best implications for improvement. For me, it's more the cases where I might actually want to hold the list for later that could be solved with TO_LINKSETDATA. Raffles, prize boards, class attendance (and delivering notes and materials afterwards to those who attended)...there's plausibly a lot of uses for getting the list of people present without immediately taking an action and then clearing the list.

And in all such cases, we know the data is coming in, and we know we need it later, and we have no estimate beforehand of how much script memory it's going to take up... Before LSD, the solution was the infamous "storage script" approach, and now it is doable with LSD albeit at the expense of having a script with enough memory to grab the list and iterate it into storage for later (with the added bytecode cost of the code to do so). Lucia's options help even further, allowing us to manage the amount to be moved per call, but at the loss of some time resolution (eg, LSL provides a list with no duplicate names in one call at present; making multiple smaller-sized calls with the intent to store the list for later would inherit the need to also diff the incoming list with the stored list). In that case I'd love an option to "put the data directly where it was going to go" - which also benefits from Lucia's options, since LSD is large but certainly not infinite itself.

And..yes. I imagined a TO_DATASTORE option as another alternative for handling incoming foreign data that isn't intended to be stored for later, but needs to be parsed and has the potential to be too large, in which case there'd be one datastore event fired for the full return value, with the queryID key used to fetch the size of the return data (in list length, if it's a list) and to fetch the elements of the data. Def not one datastore event per avi hehe.

And, what was the other–oh, yes! Perhaps we could have a multiwrite if this existed, but agreed, I raised the point on multidelete about "so, would this raise an event in other scripts too? and wouldn't there be lots of unmanagable ways to stack-heap overflow things with a potentially large list of keys?" That's another one I thought would be getter off returning a query ID with the ability to use that ID to iterate a results list. But..I dunno, maybe the amount of stuff held in short-term limbo waiting for us to fetch is a complicating weight on LL? Eh, for them to know :D

JIRAUSER341305 commented at 2023-08-27T15:58:38Z

When you take action on clearing the list is pretty much irrelevant. You can do it at the end of the current cycle, or the end of the next, it's still the same thing; create list, loop over list, delete list, rinse and repeat. I find myself most often keeping it through to the next cycle so I can figure out who's left in the meantime — still the same thing. You're still just storing the same temporary list in a different place, while your script slowly trundles it's way through (because scripts are slow) to update it's actual useful information, and then waits a bit to do it all over again — and so you typically find yourself having to handle TWO versions of that list for at least a short time, especially if you care about who left during the interval. Also, the whole point of it being a list in the first place, whether it's in script memory, LSD, or hiding inside an event, is simply to keep it stable while your script does it's thing. If it wasn't for that requirement, they'd have just given us: key llGetAgentKey(integer index). (Please don't, by the way. Just saying. I mean, it wouldn't crash your script, but, just no.)

However… The biggest issue with your TO_LINKSETDATA idea, as I read it, is that it puts an unstable (and mostly useless) value in the key, with the information (uuid) in the value. While an obvious format, that's also exactly backwards; verylongkeywithallthestuff="1" is a rather common LSD layout, primarily due to LSD's ability to search in keys but not values. From both a stability and searchability standpoint, you want your most important and stable information in the key — that'll generally be either uuid or username here (and also why it's called the "key") — you then put ancillary and changing information on the value side (in this case, you don't have any, so would have to just stuff it with a "1" or something, which seems to be the custom — I guess because "1" casts to TRUE…?). And it's that instability in your proposal as it is, which leaves you stuck in a rigid cycle of consuming the entries, storing the information you actually need elsewhere (typically making/updating a second list from the first), and then wiping it again ready for the next pass. Worse still, because LSD can't search values (hopefully that'll be addressed sometime), putting the important information (uuid) in the value part, means you can't actually find it in the list! So for every key you want to compare against the last check, you have to go through every entry one by one searching for it (it's not so bad if you can run it the other way, but that's not always what you want).

My strftime-like version of your suggestion simply lets the scripter decide what (and how much) information is important to them (including simply doing exactly what you proposed if it does happen to fit their needs), and on which side it should go — rather than just being handed a different (and even more opaque) unstable blob from which to fish out what they need. If we do get your TO_LINKSETDATA suggestion (and I don't actually object to it, despite finding it essentially redundant), I hope it'll at least be in the more useful form I suggested.

My other suggestion — the same one Lucia wrote up in that Jira I'd forgotten about (and albeit missing a fairly important detail much to my dismay at the time, doubly so because it was already "Accepted" before I had a chance to comment) — completely avoids the problem entirely (if implemented and then used right — both of which are easy, but non-obvious). The script simply doesn't need to store the temporary list at all, so long as it has a stable cursor into the actual list already being kept by the sim (in this case, a timestamp). Also, to use the "limit the number of responses" option effectively, you also need to be able to determine the timestamp of the most recent response in the returned set (and hopefully they get returned in time order so you don't have to go fishing for it, too). The the omission in Lucia's proposal, is what that easy but non-obvious detail should be; I'm hoping LL make the thresholds both be the start of the second, it's the least annoying of the three sane options. It did also just recently occured to me that so long as their join time was stored as a frame number, you could use llGetTime as the filter too with some minor caveats, like a decreasing resolution past some range (at least a couple hours, probably more like a hundred hours if my quick estimate is on the mark), and even accept both (since one's a float, the other's an integer, and it's being passed through a list).

JIRAUSER332931 commented at 2023-08-28T05:39:18Z

I think there's a really valuable point in there with searchability!

Having the capability to send the data to a linkset from the simulator without touching the script memory has, as I think we've seen here, specific and limited uses, since any immediate use is better served by Lucia's Accepted feature request, which can both fetch a reasonable amount, and an amount matching the sort of filter we might want. To that end, the primary feature here is the ability to be able to store a potentially large dataset fetched from outside the script for later analysis and use. To that end... I will modify the request in deference to the extremely valuable point about searching.

The key cannot be just a UUID because it'd be impossible to reasonably know how to clear the portion of the LSD we need. But, options flags to the rescue!

As before, [TO_LINKSETDATA, string key_prefix] could send the results directly to LSD, bypassing the need for the script to take it all in and send it all out itself...but the prefix is an identifier for the key rather than a prefix to an index number. As such, it would still be simplicity to seach for a given UUID, and trivial to iterate the found set of all keys with that prefix, and to delete any as well. And for the value, the constant 1 is as good as any, but I'd go so far as to coopt Lucia's options if supplied, as relevant. EG, if the option parameter AGENT_LIST_REGION_ENTRY_START_TIME is also supplied, then the list is filtered as per BUG-232312 and the value is the unix time they entered the region. This would supply additional utility as attendance-taking etc.

Hopefully it's ok–I'm going to update the OP to indicate this better suggestion of key naming.

Spidey Linden commented at 2023-08-30T18:10:11Z

Hello, and thank you for your feature request.

Incoming suggestions are reviewed in the order they are received by a team of Lindens with diverse areas of expertise. We consider a number of factors: Is this change possible? Will it increase lag? Will it break existing content? Is it likely that the number of residents using this feature will justify the time to develop it? This wiki page further describes the reasoning we use: http://wiki.secondlife.com/wiki/Feature_Requests

This particular suggestion, unfortunately, cannot be tackled at this time. However, we regularly review previously deferred suggestions when circumstances change or resources become available.

We are grateful for the time you took to submit this feature request. We hope that you are not discouraged from submitting others in the future. Many excellent ideas to improve Second Life come from you, our residents. We can’t do it alone.

Thank you for your continued commitment to Second Life.

secondlife / jira-archive

[BUG-234317] Add LSD option to llGetAgentList #11265

How would you like the feature to work?

Why is this feature important to you? How would it benefit the community?

Links

Related