Poor performance w/ Refined Storage

[deusversus]

I'm experiencing poor performance. A lot of lag when taking items in or out of the Refined Storage system; I have a lot of EMC (60 billion) and maybe a few hundred EMC-recipes in my tablet; it seems like the lag is coming from the mod trying to update item quantities? I don't know. I just know it's basically unusuable for me because it freezes my client whenever I try to use the system.

Did some testing in a new world. With a few EMC-recipes in my tablet, it works fine. But if I throw the Tome of All Knowledge in my tablet, the system freezes for up to a minute.

It's only w/ Refined Storage though; AE2 system lags when I first open the terminal, but afterward it works as expected.

[doofus9341]

I've experienced this same issue. When only a few emc items have been learned it works perfectly, upon using a tome of knowledge and extracting or inserting an item into a refined storage grid, the game will freeze. I however suspect it may actually be an issue with refined storage or projecte because another similar mod exhibits this exact issue but Applied energistics storage systems do not. The mod in question is equivalent storage https://minecraft.curseforge.com/projects/equivalent-storage

[pkmnfrk]

So, I haven't been ignoring this bug. Well, actually, yes, I have been. Partly because I've been working on something else, but also partly because this is a really tricky issue.

AE does work very well, and is what I tested the mod with. But, in spite of that, there is no special API or anything that I'm using. I'm just using stock Forge interfaces (albeit in a way that most storage containers do not), and AE handles it correctly.

Refined Storage, on the other hand... well, I haven't done any digging myself, but I know that other mods with excessively large inventories (Colossal Chests come to mind, from my time playing Sky Factory 2) also cause significant lag. This suggests that the problem is with RS, not EI.

That said. It is possible that Refined Storage has a non-standard interface or capability that I could use. If so, then I'm open to adding it, even though it is a custom integration. I will just need time to investigate.

[pkmnfrk]

So, I started down the path of adding explicit Refined Storage compat, but it wasn't actually helping. And, I also ran into some other issues as well.

Instead, I added an option to impose a cap on the number of items the system will expose. So, if you set the cap to 1 million, and have 1 billion EMC, it will still only show 1 million Cobblestone, etc. That way, it won't have to update every item every time anything changes.

Can you try this build, and see if it helps? https://drive.google.com/open?id=1Z-Va5dJW-olRnObiBYhgcR1rB1k-LuC2

[deusversus]

Loading it into a test copy of my world now, will report back shortly.

[deusversus]

Okay, so I did some testing. Abridged version is the fix seems to work as a stop-gap measure but perhaps not a permanent solution. Capping the items the system exposes reduces lag over-all, and on my survival server (e.g. about 200-300 learned EMC-recipes) it eliminates the lag using the RS stystem. However, I used my wife's account to learn the Tome of All Knowledge, and with so many EMC recipes, the system is still heavily burdened when pulling items in or out (refreshing available item values).

I'm not sure how many real-world scenarios will involve people having as many EMC-recipes as my server does w/ the Tome of All Knowledge (I've added EMC values to a LOT of items) and I think the 1-mil cap at least makes the mod usable for most survival settings. I might suggest making the 1-mil cap configurable (from say 64 to 1-bil) which would let pack maintainers decide how many items get exposed; that way if the ToAK is craftable, they could lower the exposed item cap as needed.

Either way, thanks for the great mod and the excellent trouble shooting. You're a life saver.

[pkmnfrk]

So, a few things:

1. The cap is already configurable, either via the config file (for SMP) or the Mod Config system (Escape -> Mods -> Equivalent Integrations -> Config) (for SSP). You can set it anywhere from 64 to 2 billion and change (before now, it was effectively capped at 2^31-1)

2. I'm not entirely sure I can make it any better. The trigger for the problem is, as you guessed, the changing of the item quantities when EMC changes. However, my code is actually pretty fast. The part that's causing the lag is something inside Refined Storage. On my test environment, with the Tome of Knowledge, I had 942 items in the system. This triggered 942 updates whenever anything changed. In my code, that took about 1ms (+/- 50% depending on the specific algorithms I was testing) to calculate, which is well within the 50ms time limit a tick has. But, the lag still happened.

   More than that, the lag froze the UI, which suggests that the lag is actually on the "client", not the "server", where my code runs (Even in single player, it's still a client/server model. That's how "Open to LAN" works).

   With that in mind, the cap is the only effective solution that I can think of. Since any item that is above the cap doesn't change any more, it will reduce the number of items that change on the regular, thus reducing the lag.

   That said, I saw lag when ~900 items were being updated. If you have a lot of mods installed, I can see you having thousands of items, and even if half of them were being filtered out by the cap, that's still thousands more that aren't.

   I sympathize with the problem, but there's only so much I can do to work around Refined Storage.

3. In case you were wondering, the reason AE does not lag is because it doesn't try and play any funny games with attached inventories. It takes at face value the fact that this weird chest has 4000 slots in it with 500 million items in each one, and just rolls with it. Refined Storage has a whole API dedicated to updating it's internal cache of items, which is a little crazy (in my opinion)

[pkmnfrk]

I'm going to close this for now, but I'm going to keep thinking about this, and maybe a novel solution will present itself to me.

[pkmnfrk]

Out of curiosity, @deusversus, were you only experiencing the lag when you had the terminal open? I think I've figured out where the lag is actually coming from, and it's not pretty.

Basically, it seems like Refined Storage is sending a separate packet for every changed item, which is why updating a bunch of things at once is so bad.

[pkmnfrk]

(see raoulvdberge/refinedstorage#1980)

[deusversus]

Yes, the lag is only when the terminal is open. I think you've hit the nail on the head here.

[pkmnfrk]

Ok, so, good news: I've fixed the problem at it's source. Could you try this (very unofficial) build of Refined Storage?

https://drive.google.com/open?id=1SjR97UMn2H3LPLhM4ZpfI_J2IMSBZ4sU

It should work with either the previous or pre-release version of Equivalent Integrations, but it may have other changes that I'm not sure about. Definitely only test in a backup or test world.

[pkmnfrk]

This should finally be fixed in this somewhat more official build: https://jenkins.raoulvdberge.com/job/refinedstorage/283/