Open gdelazzari opened 7 years ago
So, let me explain what I had in mind for the API (btw, nore on IRC is me):
The first part is the design goals for the API we need. The main objective of that API would be to focus on something that can have enough performance, because we some servers are really suffering from the slowness of the current implementation of technic. However, networks need to function in an "all-or-nothing" fashion, even if only some parts of them are loaded.
In particular, that means the following:
Now, for the implementation, keeping efficiency in mind:
I feel we should however really have a polished API before implementing it, so that we know what we want. This rewrite could also be a good occasion to shift to a more proper energy system inside of the non-sensical "EUs" we currently use, which are both an unit of power and energy (!). I'd like some input from other technic developers or from players/server owners if they have a specific request.
Anyway, all in all, this is a rewrite I've longed to do, and an unified energy system is definitely something we need to have around, so thank you for wanting to help out!
For the configurable step time thing, it might be a good idea to stagger network processing instead of doing it all at once. For example, assuming a Minetest step time of 0.05s, instead of ticking all the networks every second, you put the networks into 20 buckets and process one of the buckets every 0.05s step. This would avoid having lag spikes at the network update interval.
I haven't checked what the test implementation does, but since all connectivity info is in Lua-space, I feel like it might be cheap enough to just do a graph traversal on a network every time a node is removed from it (addition is easier to handle). Maybe that is what @Ekdohibs means by a basic data structure.
Also there might be crashes or unreported changes to the map that cause inconsistency between the table and the map. To make the system more robust, in case of unhandled deletions, it would be a good idea to pick a few network nodes every step and make sure they are consistent with the table data. This should be chosen to have good (map) cache performance. You could keep a mapping from mapblock -> set of network nodes to choose nodes that are in the same block. Unhandled additions could be mitigated by LBMs on the network nodes.
First off thank you for the interest. Regarding the network structure handling/storing, I should say that I actually have little experience with graphs in computer science, so things like dynamic connectivity and heaps (which are different from the RAM heaps I know :D) are new to me. This is to say that I'm probably already using something similar to handle the networks (at least looking at the wiki suggests so). Anyway, @Ekdohibs may eventually help me out understand this if needed, I hope :D
Another quick thing before diving into the main one. You arised the "EUs" question. I was expecting someone to do that. I totally agree with you that they are non-sensical, like EU/RF/Tesla in Minecraft for instance (electronics guy here) but the thing is... I've been following a ton of youtubers playing tech modpacks in the past (I'm talking about Minecraft) and I really noticed that the majority of them (around 80%) didn't like approaches using something more realistic like Volts and Amps. EU/RF/etc is just simpler to understand and somewhat more straightforward to use, carry around and play with. Also, I think we can consider "energy/power units" somehow like Watts, with magical wires and machines that can transport them without worrying about Volts and Amps. Anyway I'm totally open regarding this, I think we should understand mainly what the players want (this is obviously a point of discussion/voting). In addition, units like that are probably easier to handle in code.
Now into the main point: the callbacks thing. I like the idea that @Ekdohibs proposed (using a callback when a machine needs
For @raymoo: I like your idea for the network processing split. Also, yes I'm going graph traversals when nodes are removed and yes, doing some timed checks for consistency is mandatory. With the current implementation it is also quite easy to do that.
So, in conclusion, I think we should first off define what energy unit we want to implement and then, once we have that, talk about the interface that the API exposes. At this point, I would say that we could start from that i.e. we build up an example of how the API should be used (what methods to call and when etc...) and then we define and code the underlying system, collecting suggestions for performance.
One thing to consider about splitting up the workload: don't just go by network, be willing to split any given network into small steps also. I say that because my Dreambuilder survival server (daconcepts.com 30001) has a server-wide public HV electric grid.
It's currently one massive grid, kinda haphazardly-laid-out, but it's a single contiguous network with about 13 MEU of capacity (when all power sources are operating).
Adding a cable junction/spur to the HV grid, or in some cases adding a machine to it, triggers a "recalculation" of the grid that can run for around 30 seconds, during which time the server is basically unresponsive.
You'll need to take that sort of use-case into account.
As for what energy unit, I can't think of anything that's quantifiably better than an EU in a game setting, except perhaps megajoules or kilowatt-hours. The "voltage" can still be a unitless LV/MV/HV.
Of course that would surely prompt someone to want to measure continuous draw in kW for machines and generators and such, and use kW/h only for stored charge e.g. battery boxes.
(or whatever the equivalent measures are for joules)
a common API […] that will make energy distribution and item & fluid management uniform and easier for anyone willing to write mods with such requirements.
I wonder how you want to transport liquids like this since they don't move nearly instantly.
kW/h […] (or whatever the equivalent measures are for joules)
J=W*s
Btw, @HybridDog has already done something on a separated branch.
@VanessaE with the current implementation, adding a device to a networks always takes the same time, which is equal to "instant". When you place a machine/generator/battery/something, it simply looks for an adjacent wire, looks at the wire metadata to figure out what network id it belongs to, and then the device gets registered to the network by adding it to the network table. So problems like the one you wrote wouldn't be there.
Also, I would like to report some limitations that the system I have implemented has (that I forgot to mention before):
Replying to the other comments that appeared while I was writing this...
@VanessaE regarding the LV/MV/HV classification I don't know how you would implement that, probably in the mod space (i.e. not in the API) by making 3 different types of cables and programming them not to connect to each other, and then having converters once the second point above is implemented (which is, I think, how you are already doing this). But the API will know nothing about LV/MV/HV. When I started thinking and writing the system I had in mind something like RF in Minecraft, where if a machine is really big and needs a lot of power, it just asks for a lot (100000+ units/tick). The wire tiers would be the amount of energy they can carry. The progression in the game wouldn't be broken I think.
@DS-Minetest as far as liquid and items are concerned, as I said at the beginning, I am currently not interested in this. I'm focusing on the power delivery at the moment. What I meant is that, once we have a common API/system (if we'll ever have :D) throwing items and fluid management would be nice so that other mods would be able to connect to wires, pipes and tubes of any other mod for Minetest. My idea is that it would be nice to have something somehow similar to the Minecraft Technic experience, where all the mods interact with each other. If someone has already done some work regarding fluids it's nice because we can just integrate this and make this available to every mod through the common package :)
@Ekdohibs I ran a quick benchmark by copy-pasting the distribution algorithm and the few lines strictly necessary (function to register devices etc..) on a separate file and assigned 1000 users, 1000 providers and 500 storages to the network. To run 1 million cycles, the program takes 78 seconds using LuaJIT, which translates to almost 13k cycles per second (AMD FX-8350@4GHz on Linux). Seems pretty good. Obviously there was no metadata access on the storage devices (I wasn't even running inside minetest) but even if there was, I don't think the performance would be lower than 2/3k per second. Considering we need 20 cycles per seconds.. I don't see any problem here. And this is using the callbacks-every-tick approach you see in the Pastebin. Now I'm going to try implementing something like what you have described and see how much the performance increases. If it's more than 25/50 times faster I might go with that :)
Finally.
So, I have nothing against EUs (Energy Units) as long as they are accompanied by PUs (Power Units, EUs per second). Joules and watts are too realistic, you would have to define ingot mass in kilograms, for example. And on volts and amps... as long as you consider conductors and insulation ideal, these values don’t matter except of that voltage must match (also remember that IRL there are DC, 1-phase AC and 3-phase AC systems, and some others; also, at least 2 wires are needed virtually always).
What I am against of is the implementation. While the API, being suboptimal, may be polished, the implementation must be dropped. It will have catastrophic problems on real servers, for various reasons. I do understand it is here just as an example; I might do the same. Just don’t build real implementation on that.
Also, how would you join 2 networks temporarily (like in a switch)? That’s often handy with technic
, but my be useful with any such mod.
P.S. @DS-Minetest J=W⋅s, not W⋅h
While the API, being suboptimal, may be polished, the implementation must be dropped. It will have catastrophic problems on real servers, for various reasons.
Why are you saying that? I mean, what are the reasons? Help me understand :D if we can come up with something better, that's great. What I posted is just an experiment, I am ready to write everything from scratch :)
Also, how would you join 2 networks temporarily (like in a switch)?
I can come up with a couple of ways to accomplish this:
Regarding the energy unit, I think the simplest thing for the players that will make total sense to them is to define a unit of measurement (like EU, PU, RF, T, J, whatever) and to measure in/out/exchange/transfer rates (like the max rates of the cables, the power used by a machine or the power generated by something) with the same unit followed by "/t", which stands for "per tick". What this unit will be, I have no idea. EU is fine for me. I wouldn't bother with more realistic concepts for the sake of simplicity of the game.
About the units: EU for energy and EU/t for power are fine with me, as long as we do something consistent :).
Concerning joining two networks temporarily: this is the why of dynamic connectivity algorithms. But I expect that if we keep all the data needed for the networks in Lua, this will not be even needed as a graph traversal on ~1M nodes in Lua only with a correct implementation should run in <10ms. If this is still not enough, I will have to implement the dynamic connectivity.
Concerning access to metadata: it is slow. Like, really slow. The why is easy enough to understand: LuaJIT can't optimize the API calls, so each switching between Lua and C++ loses a lot of time. Besides, the cost of using strings everywhere needs to be taken into account when thinking about metadata, and it is higher that you might think. But if the code doesn't need to load metadata to give power information, it should save enough, at least for now.
Finally, about minimizing the number of callbacks: as I said, we can start a very naïve implementation that actually calls the callback on each step. Then, we could move to another a bit less naïve that just keeps the number of ticks before the callbacks needs to be called again, and decrease it each tick until we call that said callback. If that is still not enough, then we can just implement intelligent algorithms (I can take care of that if needed).
Maybe devices should be able to have pure-Lua metadata like Minetest luaentities do so that callbacks have the option of doing useful work without needing to interface with C++ stuff. For example, counting down a work timer doesn't need to interact with the map or inventories.
I have created a separated repository for this. I'll probably start from scratch but first I want to have a specifications sheet that defines everything, from API interface with nodes to performance tips&tricks and, obviously, overall system architecture. Please contribute there with your ideas, suggestions and requirements. Just create issues or, even better, pull requests are greatly accepted to help me write the specifications document. If you plan on contributing constantly I can give you access to the repository.
https://github.com/gdelazzari/tech_api
Thank you!
Will edit the opening post of the issue and put this also there
So, some problems:
globalstep
interval varies, sometimes a lot, so ticks significantly less than 1 second may not be reliable. Therefore, either you use such large ticks, or pass time interval Δt to callbacks (maybe rounded to some tick size, or even measured in ticks, if you don’t trust floats; but that may only be something very internal).Note @gdelazzari, I’m really happy someone finally decided to replace that old ugly technic
power system. If you need any help, you can find me at IRC (numzero) and e-mail (see profile).
Thank you for sharing your opinions @numberZero, I'll write some lines for each point you made to ensure we are on the same line of thought :) and maybe we can figure out something together.
For further comments, suggestions, criticism and ideas, please go to the repository so we can keep everything organized as much as possible :) also @numberZero you can find me on IRC #minetest-technic (I don't see you online there ATM)
@gdelazzari On number 1:
For efficient network updates, all connectivity info must be in Lua-space, that includes network IDs. Indexing a position-indexed table will always be cheaper than fetching metadata for a position. You will need to make extra effort so that the system is robust against missed updates, which is what I suggested earlier.
@raymoo sure thing. I'm writing the basics of the specifications document right now. Maybe you can check the repository within an hour or so and you'll know what's going on :)
defining whether or not this will be useful if I develop it completely
l did it already: https://github.com/hybriddog/technic/tree/newnet And it seems to work.
in the case of positive response to the point above, defining the API
Here's the doc: https://github.com/HybridDog/technic/blob/newnet/technic/doc/api.md
define all the best practices to make this blazing fast and not overload the network or the servers CPU in the case of multiplayer worlds
You can implement caching fairly easily: fake meta (for the machine dtime), fake get_node, etc. or use mod storage instead of meta. The implementation doesn't rely on meta usage.
Running the users or producers should not need an ABM, as these don't work for unloaded nodes.
There's no abm, just node timer for the network poller (switching station). (The abm to start node timers from older switching stations can be replaced with an lbm)
There should be a way to avoid unneeded callbacks - the main use case being that while a machine is empty, it should not impact the network efficiency at all. Ideally, devices would be able to specify the time they would like to be supplied at the current power rate until a further callback to avoid more callbacks. […] Last but not least, not running on every globalstep but having a configurable step time would be helpful to mitigate the load on the largest servers.
There's the poll_interval in the network table (see doc), machines tell the switching station that they want to be polled earlier or equal after this time the next time. You need to call the producers when you call the consumers, else they don't get power.
be willing to split any given network into small steps also
Currently my implementation traverses the whole network (also in unloaded chunks). If you split it to small steps, you somehow need to ensure that producers give enough energy for consumers and the rest is put to battery boxes. You could also stop the pollers, gradually call every machine functions (https://github.com/HybridDog/function_delayer) and then enable them again. The time it took can then be subtracted from poll_interval (also, set to 0 if it's negative).
Connecting a device to multiple networks is currently not feasible.
In my implementation it works without problems, see the (unfinished, l need to remove thrift measurements because that's the player's job) switching station code. # I think you shouldn't use some real life unit. Technic machines are fairly different from real machines, e.g. there're no solenoids. I remember the time when l thought that HV uses a different EU because it's "high voltage" and so the 1 EU of HV energy would be more than 1 EU of MV energy. Also, the doc says "For now, all supply and demand values are expressed in kW.".
So then, I guess what I was starting to do is now pointless? :D
@gdelazzari No, because your system doesn't need switching stations, and it would keep more stuff in Lua than newnet does. Newnet still has to read the map for each network node, just not more than once per poll. Your system would not need any map reads when simulating the network, except for devices that are doing something that requires map interaction, and you would only need a constant-bounded number of map reads when placing network nodes. I think that node deletion would not require map reads other than what you already get from the deconstruction callbacks.
EDIT: I haven't read your code enough to know if it needs switching stations, but if you keep track of network connectivity then you shouldn't need any.
@raymoo I'm in touch with @Ekdohibs on IRC and we'll probably decide to carry on anyway, also because I like the idea of a separated package/API not tied to Technic. I really like this game and I want to see it full of tech mods (compatible among them, obviously) :)
Also, I'm willing to rewrite this API several times until the code shines and it's the cleanest I can get. We are also collecting the best practices we can come up with, which should result in something blazing fast and really efficient (maybe not on the memory usage side, since we'll have to keep all the "meta" in memory, but I guess it won't be that bad either). No access to the map is currently expected to happen (with the implementation I have in mind).
Regarding your edit, yes: I do not need switching stations and the final implementation won't need them too, but please just trash the "thing" I posted which was just some kind of an experiment (more of a big chunk of inefficient Lua code). Will start everything from scratch as soon as I finish my last exam of this summer (wednesday) and the specifications sheet is defined.
If someone wants to have a look at the almost completed code, it's on the experimental branch of my repository. Only a couple of things are missing, but the rest should be good to go. Before I start polishing this (cleaning up the code and optimizing a few things) I would like to have some feedback, so I can eventually make changes before I ship a final "product".
You can download this demo/test mod that shows how to use the API and allows you to play around with it tech_api_demo.zip. Create a new world and /giveme tech_api_demo:user, :provider, :storage, :monitor and :electric_furnace (which is an example of how to implement a real machine).
If some concepts are still not clear (regarding the energy distribution) keep in mind that, as soon as this is finished, I'll write a good tutorial on how to interface with the system and then document the internals. The code is, however, already well documented using LDoc if you want to generate some documentation.
Thank you in advance if you decide to give some feedback!
@gdelazzari I managed to cause a crash by placing 2 producers (One connected by wire, the other merely touching it) to a storage node. However I'm unable to recreate this, leading me to believe it might have to do with the timing of when exactly I placed the node
I've also noticed that placing a storage node on either side of a producer and then removing the node that the producer is sending power to causes the producer to send the power no where (Where it should start sending power to the other storage node)
@gdelazzari I caught the first error!
The error message is
AsyncErr: environment_Step: Runtime error from mod 'tech_api' in callback environment_Step() :/home/USER/.minetest/mods/tech_api/energy/distribution.lua:213: attempt to perform arithmetic on a string value
stack traceback: /home/USER/.minetest/mods/tech_api/energy/distribution.lua:213: in function 'distribution_cycle'
Any progress?
@numberZero The energy network code is actually done - i.e. in the sense that it may still not be bullet-proof (I already did quite a lot of testing but you know - weird stuff always shows up).
My work is an "independent mod", actually useless by itself but meant to be used as a library/common API, in fact the original idea was to create something that would enable multiple mods to interact from an energy point of view thanks to a common subsystem, and that's what I think I've accomplished. Also, the performance seems to be really good. Insanely good if you compare it to the current Technic energy system.
You can find the core mod/API here https://github.com/gdelazzari/tech_api along with a simple testing/playground/example mod here https://github.com/gdelazzari/tech_api_demo that shows how to use the API. Again, the core API/mod (how you want to call it - the first link) is technically completed, so it would be time to start writing and or/porting mods to it, like Technic for instance.
I did actually start to port Technic to it but I quickly realized it was a huge work since there's a lot of stuff to adapt, and was waiting for someone to show some interest and help me out. I remember there were some issues with the node swapping used, for instance, to "light up" furnaces or machines when working (since changing texture is not allowed from the Lua API - at least not at the time, I don't know about now) and swapping the entire node was not straightforward. In fact, if you check the TODO-list on my repo you can see I had in plan to think about some kind of helper methods to do exactly this. Besides that, the rest of the porting process is just a lot of repetitive work - i.e. opening up each machine file and rewriting the machine to work with the my API.
The advantage would be the performance gain an the possibility for Technic to connect with any other mod that in the future will use the same API. For instance, if I wanted to make a small mod that adds just a specific machine without having to also include power generators, power storages, etc... I could easly do that and just connect my machine to a technic network/power generator/battery/whatever.
I have created a separated repository for this. I'll probably start from scratch but first I want to have a specifications sheet that defines everything, from API interface with nodes to performance tips&tricks and, obviously, overall system architecture. Please contribute there with your ideas, suggestions and requirements. Just create issues or, even better, pull requests are greatly accepted to help me write the specifications document. If you plan on contributing constantly I can give you access to the repository.
https://github.com/gdelazzari/tech_api
Thank you!
Hi everyone, I've been experimenting and sketching out a new energy distribution API with the aim to create something that, in the future, may become a common API (i.e. a mod that other mods will depend on) that will make energy distribution and item & fluid management uniform and easier for anyone willing to write mods with such requirements. For the moment I'm focusing just on power. Also, I'm not using metadata to pass energy through networks so the Technic mod would benefit from this in terms of performance (given the current implementation). I already talked with nora on IRC #minetest-technic and he told me to write down here my progress and my current results so everyone can possibly contribute ideas or, maybe, help with the code.
This is what I have so far. The system uses a table (under the API namespace) to store the energy networks. A network is a collection of devices connected by a transporter (i.e. a wire - a transporter is just a more general concept of something that can move energy around). A device can be a user, a provider or a storage. I think the names are self-explanatory, but if you read the comment at the beginning of the Pastebin below you'll get more information. This means that every time a player places something that is power-capable, the table must be updated. This is not so easy to manage (especially when you need to merge two networks because a wire connected them together or, vice-versa, you need to split them etc...) but the advantage is that in one minetest globalstep handler we can manage the power distribution for all the networks in the world. Also, I'm using callbacks to pass energy in/out from devices, so no metadata is used in the process (except if the machines have an internal buffer and need to save the energy they contain, or in the case of storage devices like batteries. Maybe there's another way of doing that, but I'm not aware of any at the moment).
From a "mod-that-uses-the-API" perspective, I want to keep the usage really simple. This is what I have so far (please note that everything is experimental and thrown there, there are a few things that could be implemented differently so that it's simpler to use the API, but I need feedback regarding this). This is a Pastebin with an example API usage and some comments to explain what's going on: https://pastebin.com/smDzZ6Nq
As you may notice there are two callbacks for each device every tick of the server, which is not the best thing in terms of CPU usage. However I plan to find a better solution. nora (I still don't know your username on GitHub, sorry :D) on #minetest-technic has made a proposal for this, maybe he can report it below so it's here for reference.
If you have spare time you can look at the current API implementation (which, by the way, is a mess). It is also incomplete (still no wire removal thus no network split) and has a ton of bugs, especially on the power distribution algorithm that was written in something like 15 minutes just to test out if the power was somehow getting from a point to the other (however I promise your PC won't explode). If you want to try it on a test world sure you can, just /give yourself some "unitech:user", "unitech:provider", "unitech:storage", "unitech:wire" and, if you want to get fancy, some "unitech:solar_panels" (featuring textures stolen from Technic - don't hate me). They're an example of a solar panel implementation that uses the API. Every freaking thing is "logged" on the chat, so you should be able to understand what's going on internally.
Just download the ZIP file attached and install the "mod". unitech.zip
The nodes are simply demo-devices represented by colored cubes with some text on them, just to let you know what you're doing. Please note that the API, ideally, would not define any node: other mods will do that. The advantage is that we can have a mod just providing solar panels, a mod providing fancy wires, another one providing super-useful machines etc... all working together seamlessly. The nodes I have implemented are just to test out the system. All the implementation is in the script unitech/power/power.lua
If you notice strange power distribution (e.g. storages not charging correctly, power losses etc..) don't worry, it's just because of the sketchy algorithm - which is not the priority at the moment. The priorities at the moment, IMHO, are the following:
I hope this is somehow going to be appreciated, I'll be glad to have some feedback along with ideas. If anyone wants to help me write this, they will be totally welcome :)
Thank you for your attention!