Global Item Toggling/Streaming Per Virtual World

[IstuntmanI]

Before reading this, let me tell you about what happens when we have lots of items in different virtual worlds in the same place: I, for example, have ~29.000 objects for now (every player can add up to 300 objects in his house) in the same place in a lot of different virtual worlds. When a player is in that area, the streaming time (Streamer_GetLastUpdateTime) increases a lot because there's no such thing as per-world items streaming or entirely toggling off an object for everyone.

So here's what I thought that may help us: 1. Global Item Toggling ? (this and 1.1. are better than 2., I guess) We have:

native Streamer_ToggleItem(playerid, type, STREAMER_ALL_TAGS id, toggle);

But I think that we should also have a global-variant (for all players, without being able to enable it again only for one player with the native I mentioned above) of toggling, if it is toggled off, it should be entirely removed from the streaming map (or what it is internally). It would be much better than destroying and recreating over and over again.

It should look like

native Streamer_GlobalToggleItem(type, STREAMER_ALL_TAGS id, toggle);

1.1. Item Groups If 1. would be made (or even maybe without it, this thing seems really useful), I think that we should also need Streaming Groups (well, they could be really useful even without 1.):

native ItemGroup: Streamer_CreateGroup( ); // returns and creates the first free group ID
native Streamer_RemoveGroup( ItemGroup:groupid ); // returns 0 if wasn't deleted (probably doesn't exists), 1 if it was
native Streamer_IsValidGroup( ItemGroup:groupid );
native Streamer_AddGroupItem( ItemGroup:groupid, type, STREAMER_ALL_TAGS id );
native Streamer_RemoveGroupItem( ItemGroup:groupid, type, STREAMER_ALL_TAGS id );

We could also have more efficient things with internal looping, for example:

native Streamer_DestroyGroupItems( ItemGroup:groupid );

These groups could be really useful in combination with suggestion 1. :

native Streamer_GlobalToggleGroup( ItemGroup:groupid, toggle );

For example, we could create one group for all houses and add each house object in it, easily being able to toggle them on and off.

There could be even more per-group functions, or even better than duplicating that many functions: there could be some functions to automatically loop through all of them:

native Streamer_GetGroupMaxIndex( ItemGroup:groupid ); // returns the biggest index in group

native Streamer_GetGroupIndex( ItemGroup:groupid, indexid, array[ 2 ] );
/* This should give us something like
array[ 0 ] - item type
array[ 1 ] - item ID*/

Example:

new maxindex = Streamer_GetGroupMaxIndex( ItemGroup:1 ), items[ 2 ];
printf( "In group 1 are %d valid items:", maxindex + 1 );
for( new i = 0; i <= maxindex; i ++ ) // each index should contain an item
{
    Streamer_GetGroupIndex( ItemGroup:1, i, items );
    printf( "Index %d: Item Type %d: ID: %d", i, items[ 0 ], items[ 1 ] );
}

(I think this looks like an entirely new plugin)

• When an item is removed, it should also be removed from any group it is in, of course.
• I know about the STL plugins, I found only 2 on forums and I don't think they are really stable.

2. Streaming Per Virtual World ? I'm not entirely sure that this may be possible. But instead of adding an item to the object map (or what it is, again) of each cell, we should have something like separate object maps for each virtual world (when there is actually at least one object in that world) for each cell.

---

Now I thought that I may create one area for each house to make the items appear only if the player is in that specific house, but AFAIK it wouldn't decrease the streaming time, as they are still being checked.

I don't think there's something else that could help me with this streaming time problem. It is increasing way too much, and the server FPS is going down.

---

This needs to be discussed I guess. Sorry if this is way too much.

[Crayder]

1. Streaming Per Virtual World ?

As in the virtual world parameter in all of the create functions?

[Crayder]

And about the groups, I like that idea but I don't feel like it fits the plugin. I don't know how to describe it, it just doesn't... xD

But if it's added, it'd be totally badass if it had all of the object setting functions. They would apply to each item. Like SetGroupMaterial and stuff.

[IstuntmanI]

Yes, the virtual world parameters, but there's a problem: if an object is in many worlds, the memory would be increased a lot for just an object. I don't think it is a good idea, I just mentioned it. 1. is way better.

Yeah, the groups are pretty badass. I know they don't fit too good in this plugin, but as long as there is no better and stable alternative, it would help a lot.

---

Those 29.000 objects are in other worlds (and another interior) in the /lsair airport. I tested yesterday night and it seems like the streaming time is increased by a few MILLISECONDS! for every player that is in that cell.

EDIT: I teleported ~40-50 players to that position and the streaming time increased to 100-200ms ! It is a lot ! Server FPS decreased to 110-120. (yesterday I even increased the streamer tick to 80)

[Crayder]

Yeah, a lot of objects can cause lag. But that number depends on your server's specs. That's why some of the latest features that I requested were added, if you use them correctly you can optimize streaming a lot.

[lleps]

I have the same problem. I think the problem is not in the API, but how streamer iterates over objects in the cell. I think this could be addressed by grouping objects internally on maps (like map<int(virtualworld), vector(items)>)`, so on each tick first check for items on index -1, then check on current player vw.

Im using ordinary per-player objects on houses, without streaming at all. I dont have more than 200 objects on each house, so it doesn't cause problems.

[IstuntmanI]

@Crayder, those new features won't optimize the streaming speed. Every object will still be checked.

---

@lleps, yeah, per virtual-world streaming would optimise the streaming speed a lot. In my server I have lots of maps in many virtual worlds. The problem with per-world streaming is that an object could be in many worlds and it would increase the memory a lot (but still, I only have objects which use only one world, or all).

If I had to switch to ordinary per-player objects I would need to store every object's details per-house, and it would increase the memory pretty much. With GVar plugin (which I use) it could be pretty inefficient, as it doesn't have any iteration thing and the stored data would be doubled if they are also created in-game. Also, I don't see any reliable STL plugin on sa-mp.com .

---

If all suggestions could be made it would be really great.

EDIT: I have a question: I checked all streamer's source files but I couldn't find what are the minimum X, Y and maximum X, Y for the cells. Also, how many cells are there by default ?

[samp-incognito]

Someone actually brought this up to me a long time ago, so I have known about this for a while. With features like per-player houses, placing numerous objects in the same place (but in different virtual worlds) seems to be a common scenario, and indeed, the way the streamer currently checks those objects isn't ideal. The obvious fix, as others have said here, would be to simply index each object by its corresponding virtual world. That would absolutely work fine if the object is assigned to "all" virtual worlds (-1) or just to a single virtual world. As someone said above, you could do a lookup on a map-like data structure to obtain all objects in the player's current virtual world, and then you could check every object in "all" virtual worlds as well.

The problem is that this wouldn't work well for single objects assigned to multiple virtual worlds. You couldn't do a simple lookup in that case, because the objects would no longer be indexed by a single integer. You couldn't (and shouldn't) index the objects multiple times rather than doing a lookup, because then you would have to iterate through them all anyway, which is what you were trying to avoid. The current implementation simply checks the object just once and determines if the player's current virtual world is part of the object's set of assigned virtual worlds. and that is probably the best way of going about this.

What I suppose I could do is combine the current implementation with the strategy of indexing objects by virtual worlds. Objects in multiple virtual worlds, then, like those in "all" virtual worlds, would have to be checked regardless of whether or not the player is actually in any of those virtual worlds. Objects in a single virtual world, though, would only be checked if the player is actually in that particular virtual world. This should dramatically improve performance in cases like those described above.

As for "global item toggling", I like the idea of completely removing items from the grid. That would indeed increase efficiency in situations where you're sure the item just doesn't need to be streamed at all. I can see the convenience of grouping items together for this as well. Extending this functionality to the rest of the API, though, would be an entirely different undertaking. Obviously, the solution wouldn't involve writing new group natives for every single existing native. That would be impractical. I could imagine a system like the one that currently exists for the data manipulation natives where you could perhaps input a group ID, an enum value that corresponds to the desired native, and any parameters you wish to pass to the native. Alternatively, at the very least, it would be very simple to just make a native that returns an array of item IDs for any group ID. That would make it quite easy to script as well. I'll just have to think about this.

I have a question: I checked all streamer's source files but I couldn't find what are the minimum X, Y and maximum X, Y for the cells. Also, how many cells are there by default ?

You can take a look at this. The cells are dynamically generated based on the current cell size (300x300 by default, so in a standard 6000x6000 map, there would be 400 cells). There isn't a limit to allow items to be placed outside of the normal map boundaries.

[IstuntmanI]

Your idea about the virtual world streaming is pretty fine, at least for me. All I use is either -1 or just a virtual world. It seems to be the optimal solution.

Group ID for every existing native would be too much I guess. I think that just the ability to create groups, retrieve all group items in an array (more efficient than my idea about getting indexes) and the group parameter in that global toggling function would be enough. If we can retrieve all items in an array, then we can simply use every existing native, which is enough.

You can take a look at this. The cells are dynamically generated based on the current cell size (300x300 by default, so in a standard 6000x6000 map, there would be 400 cells). There isn't a limit to allow items to be placed outside of the normal map boundaries.

So the minimum X, Y point is -3000, -3000 and the maximum is 3000, 3000 ? (I don't see where those points are set) This means that the points which aren't in that range are added to the global stream ? I have a lot of objects outside of that range.

There should also be some more internal functions to:

• Count total cells.
• Get a position's cell.
• Get the amount of items (all or a certain type) from a cell (also from the global streaming cell)
• Extract all items types and IDs from a cell.
• Set the min/max grid points, if those are indeed the current set min/max points.

[Ryder17z]

@IstuntmanI This area is specified by the game engine San Andreas is using @samp-incognito just a generic question of curiosity, the cells can be rectangular, right?

[samp-incognito]

So the minimum X, Y point is -3000, -3000 and the maximum is 3000, 3000 ? (I don't see where those points are set) This means that the points which aren't in that range are added to the global stream ? I have a lot of objects outside of that range.

No, there aren't any limits. A new cell can be created anywhere based on an initial set of coordinates. Say, for example, that you're creating an item at (4000.0, 5000.0). (The Z coordinate is disregarded.) If you walk through that code, you'll find that this happens:

box.min_corner()[0] = std::floor((4000.0 / 300.0)) * 300.0; // 3900.0
box.min_corner()[1] = std::floor((5000.0 / 300.0)) * 300.0; // 4800.0
box.max_corner()[0] = box.min_corner()[0] + 300.0 // 4200.0
box.max_corner()[1] = box.min_corner()[1] + 300.0 // 5100.0
Eigen::Vector2f centroid = boost::geometry::return_centroid<Eigen::Vector2f>(box); // (4050.0, 4950.0)

The minimum corner would be (3900.0, 4200.0), the maximum corner would be (4800.0, 5100.0), and the center would be (4050.0, 4950.0). The center is effectively the cell ID.

There should also be some more internal functions to:

Count total cells. Get a position's cell. Get the amount of items (all or a certain type) from a cell (also from the global streaming cell) Extract all items types and IDs from a cell. Set the min/max grid points, if those are indeed the current set min/max points.

I suppose that would be possible just for diagnostic and/or informational purposes. Streamer_GetNearbyItems which @Southclaw added recently would probably be more useful if you actually need to perform a spatial query.

@samp-incognito just a generic question of curiosity, the cells can be rectangular, right?

They're square to ensure that they're contiguous, so no. If they were rectangular, some might overlap.

[Ryder17z]

@samp-incognito in some cases rectangular areas would be more efficient perhaps. hence that thought.

I.e. if you have a lot of streamed content in San Fierro but only a minuscule amount in the rest of the world

[Southclaws]

That would be more like a rectangular voronoi treemap, which I believe is used in the original Doom engine's BSP streaming algorithm (or something similar). So your highly populated areas would be given a higher weight value which would correspond to a higher density/coverage of smaller cells. Though I don't think that level of complexity is required in the context of this plugin.

[samp-incognito]

There are several different spatial indexing methods (quadtrees, octrees, R-trees, to name a few) that essentially partition space even further. This wouldn't likely be of any benefit to us, though, because items tend to be more sparsely populated in this game. Plus, every item has a configurable streaming distance, so a wider net has to be cast to capture them all in a particular area.

I.e. if you have a lot of streamed content in San Fierro but only a minuscule amount in the rest of the world

The grid system is already sufficiently optimized for that. If no objects exist outside of San Fierro, then nothing will be checked for players outside of that area.

[Ryder17z]

@samp-incognito minuscule =/= nothing. Otherwise i see your point.

[IstuntmanI]

I think that these squared cells are ok as they are now, there is no need for such a thing like a voronoi treemap.

(good thing I changed the streamer tick rate from 50 to 80)

I think this issue should have a higher priority, as it will surely benefit a lot of servers.

[samp-incognito]

This should indeed be addressed, although 6e6a09a8eb3cf3195d552c2d52ac24e2a7024804 (#104) might help matters now as well.

[vojthas98]

I have a question. I have a few thousand race track objects. Currently, I have them all in a single VW, as they are used in only one place (racing, obviously). But I want to add a different type of race, which would still use the same racing tracks, but the players would need to be separated with virtual worlds otherwise they could see each other when both races were active. The only solution I can find now is assinging the objects to multiple VWs so they can be seen on both races and only there. But if this problem described here (with streaming in multiple VW's) I'd like to make it as optimised as possible. I have a map loader which could help me to store the object ID boundaries (min id and max id) for each race track and add virtual worlds to each object when the second race is activated (the second race can be in multiple VW's, since it can be started individually by players and many of them can occur at the same time). But this approach is problematic, I'd have to loop through the objects etc. but in the end, the objects would be at mosf ot the time in only 1 vw, and only in multiple vws when it's needed. What would be better (not in terms of coding because it's not any problem for me, but in terms of optimisation)? Specifying like 100 vws for each object during creation and then just assigning players to one of them, or adding new VWs to each object when the race is started? If there wouldn't be any difference then I'd just go with multiple VWs during creation since its much easier to implement.

[RonaldoBueno16]

What is the best way to transmit multiple objects between multiple virtual worlds?