Machines are unable to accept or process certain recipes

andriihorpenko commented 1 year ago

Issue description

Introduction

Frequently, Thermal Evaporation Plant won't accept liquid whatsoever after server reboot. If you place bucket inside an input slot in GUI, it immediately rejects it and places back into a cursor (video recording below will show exactly how this looks like) If you try to inject liquid via Thermal Evaporation Valve, liquid will simply not get inside the plant.

This happens frequently, almost every server reboot.

Important things to note

this affects every Thermal Evaporation Plant on the map
reforming a multi structure does not solve the issue (cab be seen on a video)
the only way to solve this issue is to reboot a server a few times until the issue vanishes
there is no error or warning regarding this issue in logs

Steps to reproduce

There is no special reproduction steps and this happens not intentionally.

As I mentioned above, slot inside a GUI rejects a bucket and puts it back into a cursor. I suspect this is some filtering issue.

Of course, I cannot confirm that Mekanism itself is acting up here, it might be some incompatibility or bug. It's just that "slot rejecting bucket" thing makes me lean towards Mekanism.

https://github.com/mekanism/Mekanism/assets/13716167/12d34a5e-b22f-416c-86a3-51ef6091dbeb

Minecraft version

1.19.2 (Latest)

Forge version

43.2.17

Mekanism version

10.3.9 (Latest)

Other relevant versions

No response

If a (crash)log is relevant for this issue, link it here: (It's almost always relevant)

No response

pupnewfster commented 1 year ago

Could you please provide the server's latest log from after it starts and when this issue is occurring? I am curious if any errors are showing in it (either with something causing the recipe to fail to load, or with some exception just being printed to the console).

To be thorough it also would be helpful to have the client's log after they try to place something in the slot as maybe the client is erroring for some reason while doing so.

Also to confirm if you look at the recipes in JEI can you see the water -> brine in the thermal evaporation plant recipe? (This should confirm if it actually loaded or if for some reason the recipe is just not present)

andriihorpenko commented 1 year ago

Sure thing! I've collected server and client logs, hope this helps us locate the root cause.

Things to note

Time of my interaction with the Thermal Evaporation Plant: 21:50
JEI always has Thermal Evaporation Plant recipes, even when the issue occurs, so it looks like they are registered correctly
The issue is present right after server is booted and first players find Thermal Evaporation Plant not working
The issue blocks both GUI interaction (bucket bounces back into a cursor) and valve interaction (pipes are unable to push liquid through the valve)

Client log (freshly started client)

latest.log

Server log (been uptime for 3 hours)

latest.log

andriihorpenko commented 1 year ago

Small update: the issue also affects Electrolytic Separator. It appears that Thermal Evaporation Plant and Electrolytic Separator break in turns or simultaneously. Let me know if I can provide anything else, this issue is causing quite a hassle.

thiakil commented 1 year ago

You appear to have one or more mods crashing during the Attach Capability phase(s), looks like CyclopsCore.

This would explain these things not working

As for pipe issues, if you're using pipes from Mekanism-Evolution, you'll need to report to them instead.

andriihorpenko commented 1 year ago

These exceptions are thrown even when the issue with Thermal Evaporation Plant and Electrolytic Separator is not present, and affects a few Immersive Engineering recipes. I can try resolve those exceptions first so we can eliminate them as a potential cause. Regarding pipes: in my scenario pipes from any mod won’t work(tested on Mekanism, Pipez, Mekanism Evolution)

andriihorpenko commented 1 year ago

CyclopsCore developer has fixed those nasty exceptions, however, the issue with Thermal Evaporation Plant and Electrolytic Separator remains precisely the same. No more exceptions are present in logs on any of five running servers.

thiakil commented 1 year ago

Please provide fresh server logs from one with the issues

andriihorpenko commented 1 year ago

Sure, here is a fresh log with faulty Thermal Evaporation Plants. 2023-08-03-2.log

andriihorpenko commented 1 year ago

I keep investigating this issue, here are my findings. By doing remote debugging, I found out that this validator is returning false, in particular this::containsRecipe. https://github.com/mekanism/Mekanism/blob/a961bcbf0cc641b08dd594e3c72c7be3dfbd85d6/src/main/java/mekanism/common/content/evaporation/EvaporationMultiblockData.java#L115

When this happens, BasicFluidTank#insert fails-fast due to isFluidValid(stack) returning false. https://github.com/mekanism/Mekanism/blob/a961bcbf0cc641b08dd594e3c72c7be3dfbd85d6/src/main/java/mekanism/common/capabilities/fluid/BasicFluidTank.java#L179 https://github.com/mekanism/Mekanism/blob/a961bcbf0cc641b08dd594e3c72c7be3dfbd85d6/src/main/java/mekanism/common/capabilities/fluid/BasicFluidTank.java#L232-L234

This happens irregularly and I am able to reproduce it on production server with 20+ players. Do you have any idea why validator may return false? Seems like a recipe issue to me, however, JEI shows recipes correctly.

andriihorpenko commented 1 year ago

Here is some more information after further inspection. BaseInputCache contains these entries:

And this is what createKey returns from the input received.

I see the difference between ForgeFlowingFluid.Source and WaterFluid.Source. This is why ISingleRecipeLookupHandler#containsRecipe returns false, hence, why validator returns false.

UPD: here is a screenshot of a cache state when everything works fine:

As you can see, there are now four entries instead of two. Any ideas why sometimes cache is not populated with all water types?

thiakil commented 1 year ago

Do you have any creaftweaker/kubejs/etc scripts that add or change recipes? or any non mekanism mods that do?

andriihorpenko commented 1 year ago

Yes, I have KubeJS 6.1 (latest version). However, I don’t have scripts that change Mekanism recipes or affect water in any possible way.

thiakil commented 1 year ago

Mekanism-1.19.2-10.3.9.homebaked.zip

Try this build, it'll make more logs. Trigger the issue on thermal evaps, and then send the server log from startup until the issue (or shutdown).

Should be no need to use this file on the client, it should just slot in

andriihorpenko commented 1 year ago

This may take some time to catch this issue. I've installed homebaked version on 5 live servers, waiting patiently for the issue to occur.

andriihorpenko commented 1 year ago

Captured the issue: https://pastebin.com/GzFnVb8j Not sure when exactly it occurred, the server is up for ~ 1 hour. Usually, the issue reveals itself right after server boot.

thiakil commented 1 year ago

Recipe stuff looks fine.

Have you also made sure that the evaporation plants are in a single chunk? (not built partly in one chunk and partly in another)

andriihorpenko commented 1 year ago

Yes, all evaporation plants are in a single chunk, we know that placing any multiblock on chunks intersection is generally a bad idea.

andriihorpenko commented 1 year ago

New update: Infusion Factory is also affected. However, this time it does not accept even items in slots or via ME interface. Once again, infusion recipes use tags. Something is definitely wrong with the cache.

thiakil commented 1 year ago

How are you determining that the recipe is the reason? in the special version, the TEP will have logged that it failed due to that, which I didn't see any of

andriihorpenko commented 1 year ago

Not necessarily the recipes, but recipe input ingredients that use tags. As I mentioned in https://github.com/mekanism/Mekanism/issues/7831#issuecomment-1675524395, BaseInputCache is missing all necessary inputs. I can't figure out why can this even happen in the first place, but I can be sure that the issue is only with Mekanism, no other mod suffer from this. That is why I assumed that has something to do with the cache as everything points towards t.

thiakil commented 1 year ago

It load the recipes from the built in minecraft recipe system, it could be any mod you have interfering

thiakil commented 1 year ago

But as I said, your logs didn't indicate it was a problem with the cache at all (as there was no line that I saw that it couldn't locate a recipe)

andriihorpenko commented 1 year ago

I think you might be missing a point. Recipes are there, all of them. The issue lays in recipe ingredients. Mekanism’s recipes that use tags in there inputs are failing.

Take a look at my investigation above: normally there should be 4 registered types of water, but in reality there are two. Vanilla’s bucket of water has a tag that is NOT in the cache, that is why TE refuses to accept that water, as there is no recipe matching exactly that tag (WaterFluid != ForgeFlowingFluid)

You can’t see an issue in logs with recipes as recipes themselves are present. And homebacked version of Mekanism logged only cache initialization and clearing, this won’t help us understand why Mekanism sometimes adds only 2 types of water instead of all 4.

thiakil commented 1 year ago

You're misunderstanding me.

The log im saying I out that yours didn't get logged is at containsRecipe, https://github.com/mekanism/Mekanism/blob/a961bcbf0cc641b08dd594e3c72c7be3dfbd85d6/src/main/java/mekanism/common/content/evaporation/EvaporationMultiblockData.java#L115

Meaning it's not being rejected where you think it is

andriihorpenko commented 12 months ago

Okay, so now I’m pretty confident about the fact that this issue has something to do with tags.

About a month ago, we’ve changed all thermal evaporation recipes to not use any tags on the input and the issue never appeared since then.

About a week ago, as a test, we’ve reverted recipe changes back to default Mekanism recipes and guess what - 12 recorded incidents of Thermal Evaporation Plant refusing to accept water.

Not only that, multitude of other Mekanism machines are affected by this issue, and according to their recipes - they all use fluid tag as an input.

P.S. I understand that this kind of issue is difficult to investigate given that it cannot be reproduced at 100% rate, but there has to be something to help understand what’s really happening. The issue occurs once in a few server reboots and lasts til the next reboot. Seems like the issue engraves during the startup process and persists until you restart the server.

thiakil commented 12 months ago

Please post your full mod list, another mod is probably messing with things.

If you could do a binary search that would help too

andriihorpenko commented 12 months ago

This is the complete modlist: https://bytebin.lucko.me/58eyZiH2ae Unfortunately, binary search is not possible on production servers with dozens of players.

thiakil commented 11 months ago

it really sounds like you've got another mod that's screwing with tags. It would only sometimes show up due to mod loads being concurrent and therefore never exactly the same

andriihorpenko commented 11 months ago

Yet the issue affects Mekanism only. From my perspective, the only valid hint is my investigation above regarding recipe caching, but if you say that's definitely not the case (even though inputCache content varies from time to time according to screenshots above) - I don't have other clues.

P.S. IIRC, a few mods were having troubles with tags on 1.19.2, stating that Mojang has changed something to tags loading order or whatever, can that be a hint?

thiakil commented 11 months ago

The cache is just what saves us from checking the whole recipe list for every insertion. Something on your server is disrupting that lookup or messing with the cache directly Though it didn't complain about recipes when I specifically added a log entry for the evap

So it could even be some mod screwing with capabilities

thiakil commented 11 months ago

@andriihorpenko is your pack available somewhere to download?

andriihorpenko commented 11 months ago

It’s not, but I can export a CurseForge profile with mods if that’s sufficient.

andriihorpenko commented 11 months ago

This is the CurseForge profile. Use Create Custom Profile -> Import in CurseForge app to import it and download all mods. Pack 1.19.2.zip

thiakil commented 11 months ago

Thanks! hopefully I can poke at it enough to find what's happening

thiakil commented 11 months ago

that contains the mods you've got, but no configs/scripts/etc and are there any server only mods?

andriihorpenko commented 11 months ago

Scripts (as in KubeJS) cannot be shared via GitHib due to privacy concerns, but I can gladly share them through Discord as a DM. Configs are pretty much defaults, so you’re good to go with freshly generated ones. Server-side mods are also on CurseForge profile above, mostly they affect worldgen, mob capping and provide some other utilities (Terralith, InControl, Chunk Pregenerator, Clumps, YUNG’s mods, ServerCore and a few others). Let me know if you need a complete list of server only mods.

thiakil commented 11 months ago

yeah feel free to post them on discod

thiakil commented 11 months ago

I'm still not able to reproduce this, I'll need you to try with more logging from when it doesn't work (the issue happens)

Mekanism-1.19.2-10.3.9.homebaked-morelogs.zip

specifically, there should be a section like

[25Nov2023 20:58:30.658] [Server thread/INFO] [mekanism.common.Mekanism/]: initing recipes for mekanism:evaporating
[25Nov2023 20:58:30.658] [Server thread/INFO] [mekanism.common.Mekanism/]: Found fluid tag: mekanism:evaporating/brine -> minecraft:water
[25Nov2023 20:58:30.658] [Server thread/INFO] [mekanism.common.Mekanism/]: minecraft:water
[25Nov2023 20:58:30.660] [Server thread/INFO] [mekanism.common.Mekanism/]: minecraft:flowing_water
[25Nov2023 20:58:30.660] [Server thread/INFO] [mekanism.common.Mekanism/]: ad_astra:fuel
[25Nov2023 20:58:30.661] [Server thread/INFO] [mekanism.common.Mekanism/]: ad_astra:flowing_fuel
[25Nov2023 20:58:30.661] [Server thread/INFO] [mekanism.common.Mekanism/]: ad_astra:oil
[25Nov2023 20:58:30.661] [Server thread/INFO] [mekanism.common.Mekanism/]: ad_astra:flowing_oil
[25Nov2023 20:58:30.661] [Server thread/INFO] [mekanism.common.Mekanism/]: ad_astra:cryo_fuel
[25Nov2023 20:58:30.662] [Server thread/INFO] [mekanism.common.Mekanism/]: ad_astra:flowing_cryo_fuel
[25Nov2023 20:58:30.662] [Server thread/INFO] [mekanism.common.Mekanism/]: -------
[25Nov2023 20:58:30.662] [Server thread/INFO] [mekanism.common.Mekanism/]: Found fluid tag: mekanism:evaporating/lithium -> forge:brine
[25Nov2023 20:58:30.662] [Server thread/INFO] [mekanism.common.Mekanism/]: mekanism:brine
[25Nov2023 20:58:30.662] [Server thread/INFO] [mekanism.common.Mekanism/]: mekanism:flowing_brine
[25Nov2023 20:58:30.663] [Server thread/INFO] [mekanism.common.Mekanism/]: -------
[25Nov2023 20:58:30.663] [Server thread/INFO] [mekanism.common.Mekanism/]: index contents:
[25Nov2023 20:58:30.664] [Server thread/INFO] [mekanism.common.Mekanism/]: net.minecraft.world.level.material.WaterFluid$Flowing@7d45e754 : 1 recipes
[25Nov2023 20:58:30.664] [Server thread/INFO] [mekanism.common.Mekanism/]: net.minecraft.world.level.material.WaterFluid$Source@505e483f : 1 recipes
[25Nov2023 20:58:30.665] [Server thread/INFO] [mekanism.common.Mekanism/]: earth.terrarium.botarium.api.registry.fluid.BotariumSourceFluid@6d617003 : 1 recipes
[25Nov2023 20:58:30.665] [Server thread/INFO] [mekanism.common.Mekanism/]: earth.terrarium.botarium.api.registry.fluid.BotariumFlowingFluid@1561c945 : 1 recipes
[25Nov2023 20:58:30.667] [Server thread/INFO] [mekanism.common.Mekanism/]: earth.terrarium.botarium.api.registry.fluid.BotariumSourceFluid@11b74f21 : 1 recipes
[25Nov2023 20:58:30.668] [Server thread/INFO] [mekanism.common.Mekanism/]: earth.terrarium.botarium.api.registry.fluid.BotariumFlowingFluid@3d10a416 : 1 recipes
[25Nov2023 20:58:30.668] [Server thread/INFO] [mekanism.common.Mekanism/]: net.minecraftforge.fluids.ForgeFlowingFluid$Source@9ca5112 : 1 recipes
[25Nov2023 20:58:30.668] [Server thread/INFO] [mekanism.common.Mekanism/]: net.minecraftforge.fluids.ForgeFlowingFluid$Flowing@27b1319a : 1 recipes
[25Nov2023 20:58:30.668] [Server thread/INFO] [mekanism.common.Mekanism/]: earth.terrarium.botarium.api.registry.fluid.BotariumSourceFluid@3f2f5281 : 1 recipes
[25Nov2023 20:58:30.669] [Server thread/INFO] [mekanism.common.Mekanism/]: earth.terrarium.botarium.api.registry.fluid.BotariumFlowingFluid@5cac8a43 : 1 recipes
[25Nov2023 20:58:30.669] [Server thread/INFO] [mekanism.common.Mekanism/]: NBT contents:
[25Nov2023 20:58:30.670] [Server thread/INFO] [mekanism.common.Mekanism/]: -----

andriihorpenko commented 11 months ago

The new build is on all live servers. It may take some time for the issue to appear again, so I will let you know when this happens and provide a complete log.

andriihorpenko commented 11 months ago

This time Metallurgic Infuser has failed and does not accept anything in the input slot (in my case, I tried this recipe). All recipes are showing up correctly in JEI, as always (this is just for the context).

Here are the logs I gathered: 2023-11-26-1.log

Let me know if this is sufficient to pinpoint the location of the issue and if I should keep monitoring other occurrences of the issue and add them here.

@thiakil

thiakil commented 11 months ago

I think I only added the logging for single input recipes (based on the issue being a tep) I'll have a look at the log later on pc anyway

andriihorpenko commented 10 months ago

Is a new build expected any time soon? I am letting you know that we consider removing affected machines and multiblocks (including TEP) from the modpack in approx 1-1.5 weeks, hence, I won't be able to keep investigating this issue afterwards.

thiakil commented 10 months ago

why would there be another build?

andriihorpenko commented 10 months ago

I assumed you noticed there was insufficient logging.

thiakil commented 10 months ago

There's enough to diagnose the water not being a valid input for a tep, which was your initial report.

So waiting on you to provide logs from when that doesn't work. You're free to remove the mods, just means this issue will get closed unresolved until someone else who can reproduce it on 1.19 wants to participate

andriihorpenko commented 10 months ago

The issue is much broader and affects not only TEPs, but a bunch of other machines as you can see throughout our investigation. A few days ago I added logs when Metallurgic Infuser broke with the exact same issue, and it did not have water as an input. Well, if you’re waiting on nothing but TEP failure, I’ll try and catch that.

pupnewfster commented 10 months ago

Out of curiosity (and this being more of a side thing), when this issue does affect machines does a /reload cause them to start working again? (TEP or otherwise)

andriihorpenko commented 10 months ago

I will try that once the issue arises again as I haven't tried running /reload.

andriihorpenko commented 10 months ago

So the TEP issue hasn't appeared yet. However, Metallurgic Infuser issue has appeared a dozen of times. I think we should add more logging coverage, including Metallurgic Infuser. @pupnewfster, /reload has no effect.

UPD: on Dec 16, /reload did help TEP to find all recipes.

andriihorpenko commented 10 months ago

@thiakil TEP has failed today, got a log file for you. @pupnewfster /reload command DID help in this case. From the logs you can clearly see that all recipes got picked up.

Timestamps for the logs:

[16Dec2023 18:15:21.122] - Server boot, Mekanism fails to initialize all recipes for TEP
[16Dec2023 18:49:05.214] - /reload executed, Mekanism got all recipes for TEP

Log file: latest.log

mekanism / Mekanism