Applied Energistics - Vanishing Items/NBTTagData

[ray73864]

Possible causes:

• Inventory concurrency issue - likely, as it happens sporadically
• Bug in new chunk provider - not likely, as the issue can occur without reloading the chunk.

[HaoSs07]

i can give you the region for my house https://dl.dropboxusercontent.com/u/76651664/r.-6.-5.mca put in region folder in world (any world under ftp ultimate) coordonates -2794 63 -2532 remove chunk loaders. enter exit the chunk many times, bug should happen

[Stoo42]

Just to add my voice - I also have had this issue on FTB Ultimate 1.01. The chunk was always loaded with a chickenchunks loader, and also appeared to only happen to the chests. I have an ME Drive nearly full with preformatted storage, and it wasn't touched. The chests I had next to it (4 of them), had 1k, 4k, 4k, 64k storage. I could watch the chest interface and see the items slowly drain out. It seems they gradually "leak" rather than all being deleted at once.

I'm not sure the ME Drive is a red herring, as these storage were also formatted, so it might be that instead of the fact its a drive...

Definitely not an easy one I dont think, I dont envy you, nallar!

[HaoSs07]

i use world anchor, and it seems to help, chuckenchunks does not

[dantherrien]

My issue was always an immediate storage cell wipe when moving it into or out of an ME chest. It only seemed to happen when I did it in quick succession but that might just be in my head. I usually use the ME chest to see what is actually on a single disk outside of the regular terminal interface. Wiped at least 2 disks on me so I have since discontinued using it and just gone the route of IO ports + preformatting to guarantee the contents of a disk.

[Stoo42]

I haven't tried world anchor, but when you need more than the 9 chunks it provides, its a bit of a pain having to have a separate system just to keep them stocked with e-pearls!

Also noticed that pulling storage cells in and out of chests seemed to speed up the loss process. Have to admit I didn't play with it much more than this before rolling server back to the backup I took before putting TT on....

@nallar if you need some more scenario testing, I have can fire the TT world back up and try stuff for you if it helps resolve the issue, or even if you need another set of eyes cast over the code in question, I've done a bit of threaded dev before...

[Maximuscr31]

I had too many issues with the me and chunkloader crashes. when TT worked my server wouldn't drop below 20 for anything hardly. When it broke though it was a constant struggle. I am thankful for the mod and think it will be fantastic eventually. I can't begin to fathom the dedication it takes to redo everything mojang should be doing themselves. I can't wait till it gets stable but it is too alpha for me right now.

[geoffcorey]

Currently running 1465 and several of us have been experiencing items disappear from ME. I tried various things to try and replicate including loading items in and bouncing around worlds. Flooding the system. Positive it is not a power issue. However there is one thing that seems to be constant when items are disappearing.

When items are crafting and I go to another world (there is a chunk loader on the AE system) that is when things typically disappear and usually items that are used in the crafting recipe. For instance I have a recipe for Ultimate Hybrid and it was waiting on uu. Well then I fired up a recipe to make 1024 stone bricks from cobble and then bounced to another world. Glowstone, Ender pearls and blaze rods went to zero. Both the ender farm and blaze farm were running.

[geoffcorey]

Let me know if you want the backup of the world. it currently is about 2G. We can setup a chat session and I can walk you though it when you have time

[HaoSs07]

will test after we fix the teleport bug, trying to get a crash log now

[geoffcorey]

Able to reproduce with build 1481 last night.

Pretty sure autocrafting is part of the equation that is causing the problem. I am going to try and isolate the conditions in which this occurs.

[HaoSs07]

i lost items from a system that was not linked to any autocraft

[geoffcorey]

HaoSs07 do you have a lot import buses currently doing work when you lose work? It is one of the scenarios I'm going to test today.

[geoffcorey]

ok, autocrafting computer/storage was taken offline. I have a mob trap that does obsidian pipes to iron chest. Gates pulsing the obsidian pipes and the iron chests have import buses. I fired up all 3 spawners (witch, ender, wither skellie) and watched the inventory and right before my eyes I went from 500+ ender pearls to 1.

[geoffcorey]

So with the blaze rods and ender pearls I have a system with an emitter that will turn on export of pearls to an autocrafting table to produce ender eyes and blaze powder if it falls below a stack in the AE system. I watched the inventory with mob farms in full swing and boom all my eyes disappeared as well as blaze rods.

I then removed the export buses from the autocrafting tables and let the system run and nothing disappeared. I went back and added the export buses and so far nothing has disappeared.

I do have a new theory. If exporting to something that does not have room, such as an autocrafting table, the system may still export items but the recipient of the item cannot except so it is lost. I suppose this could be true for machines with relays for autoprocessing ore.

However the outlier to that scenario is the fact that other items would get lost traversing the AE system. I wonder if there is some limit on the number of items traversing AE and if it goes above that number those items get lost?

[HaoSs07]

well export bus has slots for exported certine items. i had only a disk with a chest and come cells, deuterium, trimium, something like that, i exported empty cells and deuterium, but not the tritium, no export bus for that, but i lost the tritium too. i only had left some empty cells. 10% of what i have

[LunNova]

@Stoo42

It would be very helpful if you could look into it, I just don't have the time to go through AE's code and find what's broken. I'm afraid I don't really have any idea of what's broken, so good luck if you do try :P

Thanks.

[Stoo42]

When I have observed this bug, it has drained ALL items on storage, even ones that had not been exported or imported or specified in any crafting.

its weird, import/export activity may be the trigger, but this may just be coincidence as most people with AE will be doing import/export constantly anyway.

Its almost like the storage chips themselves are ticking. I'd guess the contents of storage chips are stored in Tag Data. Because most items tag data size is in the order of a few bytes, and AE seems to be the first mod that will store up to 64kB in a single items tag data.... Maybe this is where we should be looking. Thats a lot of data in that hash table, if anything TT is doing to it to manage it doesn't take into account that the storage chips have such big tag data, then this may just be classic memory leak and not related to threading as such at all. Copy even 1k storage chips tag data to mem, but when you save it back for some reason its not saving the full thing back. In pretty much all other items in game, I cant think of a case where tag data for a single item would get this big.

This is a guess, but it accounts for why AE uses those storage sizes, and how it calculates how much is used. Its probably just measuring the size of that region of tag data for the storage chip item - I have to emphasize I haven't seen the code, this is just speculation from what I know of how items work.

If this is the case though, question is then why storage chips are ticking in the first place, or even touched by TT if they are just plain "items" - unless I have a fundamental misunderstanding of what ticking entities are in game, or what bits TT touches... :-P

[geoffcorey]

@Stoo42 you may be on to something there. I plan on pulling the world down and see if I can replicate without TT. If I can then I will try to upgrade Forge and AE on the Universal 1.0.1 and I did more testing this afternoon and it does seem to be a race condition and likely has something to do on the emitters. It appears to be some sort of race condition.

I found this on AE defect list that seems like a candidate that might be related http://ae-mod.info/Tracker/0076/

[LunNova]

I don't change the storage of tag data, other than from a hash map to a concurrent hash map. The only difference there is that it doesn't allow null keys.

I expect the issue is when two AE blocks on the same network access inventory at the same time. An easy way to confirm that it is due to concurrent ticking of tile entities in one network would be to make an AE network entirely contained within a single chunk, and not on any of the chunk's borders. If the issue no longer occurs, then that's the issue, as two tile entities in the same chunk are never ticked concurrently. On 1 May 2013 23:13, "Stoo42" notifications@github.com wrote:

When I have observed this bug, it has drained ALL items on storage, even ones that had not been exported or imported or specified in any crafting.

its weird, import/export activity may be the trigger, but this may just be coincidence as most people with AE will be doing import/export constantly anyway.

Its almost like the storage chips themselves are ticking. I'd guess the contents of storage chips are stored in Tag Data. Because most items tag data is in the bytes, and AE seems to be the first mod that will store up to 64kB in a single items tag data.... Maybe this is where we should be looking. Thats a lot of data in that hash table, if anything TT is doing to it to manage it doesn't take into account that the storage chips have such big tag data, then this may just be classic memory leak and not related to threading as such at all. Copy even 1k storage chips tag data to mem, but when you save it back for some reason its not saving the full thing back. In pretty much all other items in game, I cant think of a case where tag data for a single item would get this big.

This is a guess, but it accounts for why AE uses those storage sizes, and how it calculates how much is used. Its probably just measuring the size of that region of tag data for the storage chip item - I have to emphasize I haven't seen the code, this is just speculation from what I know of how items work.

If this is the case though, question is then why storage chips are ticking in the first place, or even touched by TT if they are just plain "items" - unless I have a fundamental misunderstanding of what ticking entities are in game, or what bits TT touches... :-P

— Reply to this email directly or view it on GitHubhttps://github.com/nallar/TickThreading/issues/246#issuecomment-17309930 .

[HaoSs07]

you guys also should check why there is no wipe when the chunk where the disks are is loaded with worldanchor . Or it happens rare,but only to the first disk ( in my case )

[geoffcorey]

@nallar that sounds consistent with what I am observing. When multiple things are running it would sometimes happen. If nothing but one thing was running it would not. I will redo my setup tonight and try it all in 1 chunk and see if there are any glitches or missing items.

[geoffcorey]

It would appear that putting the AE system in a single chunk has fixed the issue. I will continue to test the single chunk theory this weekend.

[geoffcorey]

So odd item with the test. I did this single chunk in a mystcraft age. The age is stable. In the chunk I have a power cable running the inside edge of the the chunk to the AE system. In the last few hours that entire glass fibre line went missing. I dug up the floor and not just a segment but the whole line from the corner of the chunk to halfway across the chunk including the elbow that would take 2nd block from the chunk boundary.

UPDATE: Upon further investigation there was a server crash related to someone writing bad Lua code for their turtle. Tough to pin this down. TT errors logged do not seem to correlate to this AE test system and those defect tracked in other issues.

[geoffcorey]

1513 able to reproduce the problem (baseline) 1514 able to reproduce the problem 1515 able to reproduce the problem + able to reproduce the problem of the glass fiber line connected from MFSU to AE disappearing

[geoffcorey]

Ok build 1520 it was looking real promising. No issues until I triggered an emitter on an autocrafting table in another chunk to keep 64 ender eyes and 64 blaze powder in stock. Once that happened the stuff in transient went to zero - uu, ender pearls, blaze rods. All gone.

[HaoSs07]

this problem is really annoying

[geoffcorey]

well I will say the work around of having the AE system in a single chunk works. Only issue I have had is the occasional fibre optical cable disappearing from mfsu to the AE system.

[floh22]

i know what you mean. i cant use TT on my server because of this

[wormzjl]

For the moment, use storage bus + diamond chests.

[HaoSs07]

storage bus + chest/barrel works. but the it lags the server a little when you craft hard items

[floh22]

well im cant really stop people from using those items from ae. i really dont wanna ban em either

[geoffcorey]

I'm running TT and we all moved our systems to a single chunk without issues.

[Maximuscr31]

Any update on build 1520?

[HaoSs07]

geof already posted about 1520 " Ok build 1520 it was looking real promising. No issues until I triggered an emitter on an autocrafting table in another chunk to keep 64 ender eyes and 64 blaze powder in stock. Once that happened the stuff in transient went to zero - uu, ender pearls, blaze rods. All gone. "

[Maximuscr31]

I saw that but was wondering if there was any more updated info to it besides it just dumping items with autocrafting tables. That can easily be warned against if that is the only issue left with it. I just don't go through another restore session with my world if I don't have to if I try it again and ME chest/drives emptying would cause me not to try it again.

[LunNova]

I can't guarantee that that is the only possible issue after that partial/failed fix.

[dantherrien]

Geoff and I have rebuilt both of our systems into a single chunk each and have seen no further problems. The issue was not only with autocrafting tables...seemed to be concurrent access on any inventory item across chunk boundaries that would wipe the item from your system.

[Maximuscr31]

That is what I was trying to understand. Thank you for explaining where it is at dantherrien.

[HaoSs07]

@nallar probably silly question., but have you talked with AE dev ? he sure knows his mod and may have useful tips, don't see why he wont help

[LunNova]

I have talked, but the version of AE you're on is very out of date so it doesn't help much.

The code in AE related to this has been redesigned in newer versions to improve performance, and the bug probably doesn't exist in newer AE versions anyway.

Even if it still happens in newer AE this would be a lot easier to fix, as the inventory code for AE was also cleaned up. It's already a pain understanding decompiled code when the decompiler gets it wrong, and when you can't remap it easily, and design issues in the version of AE used in FTB just make this worse.

[HaoSs07]

then probably is time to let it go and think of the 1.6 update, ftb update to 1.5 will be private

[HaoSs07]

at first look it seems to be fixed :D but will need more tests before you can safely close this

[Maximuscr31]

Now this is exciting news. Thank you Nallar.

[floh22]

hows it looking on your end haos? i havent got any problems yet

[HaoSs07]

good so far. would be nice if @geoffcorey can test too

[floh22]

well ima just add it to my server. lets see if any1 complains :)

[ray73864]

So if i stick this in, i won't be required to put all my AE stuff into a single chunk will i? because there is no way i could fit my entire AE automation stuff into 1 chunk.

[HaoSs07]

i have my system in 4 chunks

[ray73864]

And it will work on the current FTB Ultimate pack for 1.4.7?

[HaoSs07]

yes. i use ultimate 1.4.7