Storage nodes not always receiving power from providers

[cyian-1756]

I've noticed that sometimes, if you remove all nodes providing power to a storage node and then place a provider node next to the storage node, the storage node doesn't receive any power. Jugding by the info messages this has to do with the nodes being in 2 different networks

[gdelazzari]

Are you sure the provider node you're placing gets attached correctly? i.e. the side facing the storage device is one of the sides listed in the linkable_faces table that gets passed in the tech_api.energy.register_device method config table

If you're playing with the tech_api_demo mod I shared here https://github.com/minetest-mods/technic/issues/380#issuecomment-316329880 then providers don't connect on the rear side, as of line 28 in init.lua in the provider node register call to the API:

-- a wonderful provider device definition for the node "tech_api_demo:provider"
-- that keeps producing 10 EU/t (for free!)
tech_api.energy.register_device("tech_api_demo:provider", "default", {
  class = { 'default' },
  type = 'provider',
  max_rate = 10,
  linkable_faces = {'back', 'top', 'left', 'right', 'bottom'},
  callback = function(pos, dtime, request)
    local produce = math.min(request, 10)
    minetest.get_meta(pos):set_string("infotext", "request=" .. request .. "EU/t - producing=" .. produce .. "EU/t")
    return produce, 8 -- return energy produced, and ask next callback within 8 time units
  end
})

[cyian-1756]

Are you sure the provider node you're placing gets attached correctly?

After a bit of playing around it seems that I'm not

[cyian-1756]

Turns out I was wrong, I can recreate it with the provider facing any dir. I can repo this issue by placing a provider and storage node, exiting to the main menu, reentering the level, removing the provider, placing a new provider.

If I go that then the storage node doesn't receive any power until I remove and replace it

[cyian-1756]

After some debugging this seems to be causes by storage nodes not running any backs (I put a minetest.chat_send_all in distribution.lua at line 175). This debugging statement runs if I place a storage node and keep playing, but if I exit and reenter the game it doesn't.

Edit: After some further debugging it seems that storage node callbacks are the only ones not firing, user and provider fire just fine after a reload

[gdelazzari]

Try to change line 47 in the init.lua of the tech_api_demo mod to return 0, 8 so the callback timeout has a bit more headroom (I'm thinking about the callback_countdown going negative thus not firing as expected). Also note that, as the TODO list on the readme suggests, the nodestore (the module that keeps data persistent between game sessions) is still a complete draft, aka "bare minimum to test if the rest of the code works" :wink:

Implementing a good persistency solution is the next step in the roadmap, but I now lack the time to do that. This means that, till now, I didn't particularly care about what happens when you close/reopen the game. Getting an out-of-sync nodestore is pretty easy, causing quite a bit of unexpected behaviors. We need periodic backups/disk updates and a way to rebuild the entire table when we feel everything went totally wrong. This specific issue isn't probably strictly related to the nodestore but sure thing it's part of the "what happens when you close/reopen" development phase I haven't still looked at.

[cyian-1756]

After some more debugging, the storage callback seems to fire perfectly fine after a reload, so long as there is a wire connected to the storage node (The wire doesn't have to lead to anything, it just has to be there)

[gdelazzari]

Is the storage device you're playing with also registered as a transporter? (besides as a device)

[cyian-1756]

Is the storage device you're playing with also registered as a transporter?

Yup. It's the storage node from tech_api_demo

[gdelazzari]

Yeah that was probably caused by this problem in the discovery algorithm, try now and let me know

[cyian-1756]

This seems to fix the issue with storage nodes, but now the provider callback isn't firing until I remove and replace it

Also now, attaching a wire to the storage node and then removing it (The wire) stops the storage callbacks and removes the storage node from the network

[cyian-1756]

You last commit has nearly fixed this but now attaching and removing a wire from the storage node removes the provider (Which in the test set up I used is touching the storage node) from the network

[gdelazzari]

Did the last commit fix everything?

[cyian-1756]

Did the last commit fix everything?

Yup. Now the only error I can find is a Json decoding error when a network gets very large and spread out

[gdelazzari]

Json decoding error when a network gets very large and spread out

That probably is issue #2.

Thanks for taking the time to bug test the code :)

[cyian-1756]

That probably is issue #2.

That's what I was thought. I imagine that a embedded database would be better going forward.

Thanks for taking the time to bug test the code :)

No problem.

[gdelazzari]

I imagine that a embedded database would be better going forward.

Agreed, if you have any suggestions on a DB that would suit our needs, let me know. I didn't find a lot of options back when I looked at the nodestore persistency problem. Also the DB should either be fast AF (writing to disk) or some intelligent code to dump the data every x time must be implemented...

Also, if you have any ideas on how to implement a way to rebuild the entire table when it gets out of sync, help is really appreciated. I have some ideas but I feel like they're not that great.

[cyian-1756]

Agreed, if you have any suggestions on a DB that would suit our needs, let me know

Looking around I can't really find any pure lua databases, and I'm not sure if minetest would let you use something like sqlite3 from a mod

or some intelligent code to dump the data every x time must be implemented

Doesn't seem that hard, Store the database in memory, and write it to disk/update the copy on disk every X callbacks

Also, if you have any ideas on how to implement a way to rebuild the entire table when it gets out of sync, help is really appreciated

I've only skimmed the code, but what might work is to go over ever node that is a device/transporter/whatever and call either swap_node or place_node on it. This would replace the node with a copy of it's self and fire all the call backs as if a player had placed it thus rebuilding the network.

[cyian-1756]

After some digging I found https://github.com/uleelx/FlatDB which seems like it would work

[gdelazzari]

Doesn't seem that hard, Store the database in memory, and write it to disk/update the copy on disk every X callbacks

The problem is, IMHO, that we can't rewrite the whole thing every X callbacks. What if you're playing in an insanely big server (like VanessaE's one) and every X callbacks you need to dump an entire table of 10000+ blocks (with the associated data) to disk? You may even have a fast SSD but I don't think it would be fast enough. I was thinking about saving just the changes in incremental, separated files, and then every X time merging the changes into the main DB file, in order to at least reduce the interval between every main file rewrite. If the game is killed and there are unmerged files/changes, then on startup a merge happens. But IDK if this is a good idea, it adds complexity and, consequently, bugs and problems. Also a complete rewrite would happen anyway, just at a higher interval, but if writing the entire table takes time you'll notice freezes anyway. It would really help if we could go multithreaded in the API, but AFAIK we don't have threads inside the Minetest Lua runtime.

what might work is to go over ever node that is a device/transporter/whatever and call either swap_node or place_node on it

Sure, but I don't think we can iterate through the whole world and check if every node is a device/transporter and then do whatever we need to. I'm always thinking about a big server like before, where doing something like this would probably take forever. Maybe we could build an external tool in C/C++ or something faster than Lua, also taking advantage of multiple threads, to rebuild the nodestore this way, that would be the "brutal" way.

[cyian-1756]

What if you're playing in an insanely big server (like VanessaE's one) and every X callbacks you need to dump an entire table of 10000+ blocks (with the associated data) to disk?

I imagine this is where something like sqlite would come in handy (although I'm still not sure if you can use it in mods), it should be able to handle writing/updating 10000+ records fairly quickly (However I've never used it from lua, so I'm not sure about it's performance here)

[gdelazzari]

The problem is that the majority of the Lua libraries for sqlite are just wrappers to the C library, which would be useless since I don't think Minetest allows for compiled libraries to be loaded. Maybe it does, or maybe there are sqlite implementations written entirely in Lua, I'll have to do a bit of research.