Experimental async_controller (luac)

[TheEt1234]

Repo: https://github.com/TheEt1234/async_controller/tree/main

Problems:

Ratelimiting

basically solved, just need to figure out what to do if someone spams like 50 of them in a mapblock... i think mesecons_debug can figure that one out it can't

And also needs more testing

Discussion (has some important context)

starts from https://discord.com/channels/513329453741637637/719950700485935145/1214324327185649684

For those without access to Discord, the discussion mentioned above, has been added to this thread.

[Klaranth]

et i did a thing https://github.com/TheEt1234/async_controller/tree/main [needs to get seriously tested]

Huhhila no tests implemented yet?

et Do i need to do that probably... hmm well, i guess testing every single thing intended/working in the enviroment can't hurt, todo: tomorrow Also the code is dogwater, because i have no idea how to propertly share enviroment variables

et ok so i have made """tests""" https://github.com/TheEt1234/async_controller/blob/main/Tests.md and also added a metatool and the readme is actually readable now so is the code, instead of 50 arguments, i just have a table like a sane person

should i make an issue at https://github.com/pandorabox-io/in-game/issues for this though it will still require a lot more testing than what i have

now i guess i am gonna try making a survival friendly copy paste tool

SX also added a metatool; If so then isn't this already survival friendly copy/paste tool?

et i meant uh, copy pasting buildings; that wasn't related to the async luac

SX Oh well then go ahead, yeah I thought it was related to async luac stuff. I guess main concerns will be how many threads one can spawn with this thing and how it controls resource usage. But yeah, guess would be worth it to open in-game repo issue for discussion.

et and how it doesn't controls resource usage. it time outs like normal except i made the maxevents be 10x more (changable) and also burns like normal many threads one can spawn with this thing given enough async controllers, infinitely many, dont know how i am gonna limit that, maybe certain amount of async controllers per mapblock? idk

SX It'll probably still be good Proof Of Concept, resource limits for actual multiplayer use could almost sure be added later fairly easily if basic thing works well enough. If it works fine it would likely be better way even if limited similar to current pandorabox luac.

et also it is still gonna get ratelimited by mesecons_debug if you spam digiline_send i think so thats at least nice it would likely be better way even if limited similar to current pandorabox luac. It is already limited that way, except maxevents by default are 10 times bigger

SX is it at most single concurrent thread per controller?

et single minetest.handle_async call per event/luac trigger so yes(?)

SX So it can be more than one concurrent thread per controller if say you send 2 events within 1 second and execution takes more than a second, then single controller will run 2 parallel programs right?

et yes maybe that could be limited, there are 2 ways to do that: 1) delay the event (i don't know how) 2) drop the event

SX or might work well enough to just increase heat value by total utilized time and let it burn :luacontroller_burnt:

et by total utilized time in what, like increasing it by the amount of seconds it took? or miliseconds?

SX yeah, counting how long each thread used up, summing up to heat value when thread returns. that would allow utilizing overheat function that already exists and through that might also already offer good enough configuration. of course it needs some balancing but as a concept it might work well enough without having to limit threads (if thread spawn rate is anyway somewhat limited).

et (if thread spawn rate is anyway somewhat limited). Technically it's limited by 20 threads/second yeah, counting how long each thread used up, summing up to heat value when thread returns so just heat=heat+execution_time

SX basically that yeah, just have to protect measurement well enough so it is not too easy to spoof it. probably have to measure it inside the worker anyway to get accurate timings.

et so just heat=heat+execution_time oh... its gonna be more complicated than that.... because the only way to modify heat is thru mesecon.do_overheat [heat = heat+1] mesecon.cooldown [sets heat to 0] how would that be spoofed.... if its like local start=get_time() runcode() local end=get_time() local timetook=start-end

SX Basically that yes, and for spoofing it you have to make sure that whatever is inside run_code() can't gain access to local variable start

et that would be seriously bad if that could happen

SX I've not played with minetest workers, no idea how well those are protected. If there's a way to gain access to debugger then that at least must be disbled. And of course any unsafe functions that might allow executing direct instructions/bytecode which in turn would be basically same as debugger access.

et luacontroller basically temporarily restricts enviroment so that no outside things can be accessed/modified and yes minetest async does have working debug that's why i was able to do this whole thing

so when adding minetest.debug(time_took) repeat until timeout -- took around 183 microseconds

repeat string.rep("a",64000) until timeout -- around 19 000 microseconds

-- these are really random, it takes hoewer much my computer feels like

@SX how much more should the bottom one be punished than the top one, i can only increase the heat of the luacontroller by +1 (so heat can't be like 19.35+math.pi)

SX Seems like you do have default lua controller instruction limiter enabled which is probably main reason for these results, I mean first one hits that limiter before it can do much work. I guess might be good to bump up or even remove instruction limiter to test properly. Could also see what things like s = "x" while 1 do s = s .. "x" end does.

et i am testing "how much resources can an async controller use up before it timeouts" and based on that, how should i punish the async controller for consuming resources

et i should probably disallow this... yeaahh... ig i can limit the maximum amount of digiline messages sent

et ok i have done that

SX Though async has nothing to do with end results really and regular luac does the same as long as you do not limit outgoing messages.

iceyou yeah most likely it doesn't affect it but I have no idea how frog implements async functionality into it

SX Async processing in minetest is through workers that will be synced with main thread through basic locks, basically end results / responses from async function is handled just like fully synchronous single thread program.

SX There's no real threading available, just asynchronous workers that take a task, can be executed and then main thread will check it each server step and grab results if worker has finished its thing. Worker itself does not have access to world or game main thread-

et see readme, if the async luac times out or errors it will still send out the messages also i just realised i spelled environment wrong... let's see if somebody notices but i've limited it to 150/event, rest get dropped

et minetest.handle_async(async_function, callback,arg1,arg2,arg3...)

iceyou yeah functionally this tells me right about nothing

et does this tell you anything https://github.com/TheEt1234/async_controller/blob/main/init.lua#L610 and yes before there were like 100 arguments given to the minetest.handle_async function

SX That timing out or errors are simply just decision for how to proceed, basically digiline_send is exactly same as it is with regular lua controller: it will collect messages and send those after program is completed. Only thing that limits regular luac is instruction limits and error handling which just do not exist in async luac like they do in regular luac.

iceyou (Idk how minetest.handle_async even works)

et neither do i it asyncs then when the async function returns the callback gets called

instruction limits do exist in async luac, they are just 10x bigger than in normal luac i think it would be a good idea if we made a github issue for this.. or does it need more testing? and also right it needs the ratelimiting

SX Everything written here will be forgotten very soon unless it gets copied to github issue, which is why I did recommend creating one

et well, it can be forgotten, because this is mostly discussing the implementation/limits of it

SX Thing is we'll be discussing all that again when it comes to actually planning to use it on server

birdlover32767 imma try doing some suggestions

et also i changed the burnt texture to be this async_controller_burnt_top.png

@SX soo do i just create the issue and copy over some context

frogTheSecond also one thing i forgot to mention (i think), i added a print log and remade the print() and added clearterm()

SX A lot of discussion so would be nice. Additionally if there's a lot of specific implementation discussion then some of that detail could maybe also be their own issues (possibly linked to main issue) on your own repo too if you'd like to keep main idea issue a bit cleaner.

et yes, just like the normal luac (at least feels like)

[TheEt1234]

i've rewritten it a bit (seperated the init.lua file into smaller files)

[TheEt1234]

the ratelimiting problem has been solved (hard 10ms [previously 20ms] limit per event) and some of the bad luac bugs are harder to do now

and also fixed that repeat mem.i = { mem.i, mem.i } until timeout bug that made the serializer go waaa by disallowing tables like that

[birdlover32767]

i feel like the recipe is a little too... cheap because of the computing power maybe something like

mese block digiline integrated circuit digiline mooncontroller digiline silicon lump digiline bronze ingot -> 1 async luacontroller i mean you have plenty of spare ores, right

[S-S-X]

Could also make sense to use multiple lua controllers and maybe few ic's for recipe as it will be multi core in the end anyway.

Like 3 - 6 luac on bottom row(s), maybe ic or two on top of that and some copper for heat transfer (block if you feel it would be too cheap otherwise). For recipes I think it is good to think what it actually does and try to reflect that on recipe if possible.

[TheEt1234]

Sorry for not responding, i was kinda distracted with developing a new mod

I will change the recipe then, maybe to something like

(where luac = luacontroller, copp = copper block and heat = heatsink)

copp, heat, copp
luac, luac, luac
luac, luac, luac

[TheEt1234]

[SwissalpS]

I like this recipe a lot more.

Another kinda expensive item is digimese.

IIRC heatsink is from [digistuff]. Maybe you want an alternative recipe for servers that don't have that mod installed.

[birdlover32767]

digimese can be replaced with a mese block/whatever heatsinks can be replaced with a mesecon

also better textures could be used (most namely the burnt async controller)

edit: here are my textures

[TheEt1234]

(most namely the burnt async controller)

:( i thought that one was good

but seriously burnt luacontrollers to me always looked like angry luacontrollers

[birdlover32767]

fine, the new burnt texture is

ok but seriously the real burnt texture (with a face) is

[SwissalpS]

I was about to say, at 16x16 pixels none of the text will be readable.

I don't mind a frowny face, but it might be a bit over the top. Just a shade darker red might be sufficient.

[TheEt1234]

i would love if it was more different from the digistuff component textures (why are they so overused) but i have no idea how to achieve that

Also birdlover, why add that text?

[TheEt1234]

also, should the async_controller include extra stuff in it's environment like a safe version of loadstring and pcall? and also some vector stuff too(?)

(The main issue with pcall was that the execution could continue after the hook got wiped)

And loadstring with a character limit and proper sandboxing (not allowing bytecode, disabling jit, setfenv, etc.) would most likely be safe and allow for cool things

[TheEt1234]

there are also a lot of things in the minetest namespace that are safe

also idea: maybe there should be a node, that when placed next to the async_controller will allow access to this fancy environment(?)

[birdlover32767]

Also birdlover, why add that text?

that was for... april foolz

also here is a list of safe mt stuff would be good if you erased the stuff from the "gray zone" (like minetest.get_modnames)

• ItemStack
• ItemStackMetaRef
• PerlinNoise
• PerlinNoiseMap
• vector
• bit (yes, this exists)
• minetest.get_modnames
• minetest.get_game_info
• minetest.is_singleplayer
• minetest.features
• minetest.has_feature
• minetest.get_version
• minetest.sha1
• minetest.sha256
• minetest.colorspec_to_colorstring
• minetest.colorspec_to_bytes
• minetest.encode_png
• minetest.urlencode
• minetest.string_to_privs (probably with a priv digidevice)
• minetest.privs_to_string (probably with a priv digidevice)
• minetest.formspec_escape
• minetest.explode_table_event
• minetest.explode_textlist_event
• minetest.explode_scrollbar_event
• minetest.inventorycube
• minetest.dir_to_facedir
• minetest.facedir_to_dir
• minetest.dir_to_fourdir
• minetest.fourdir_to_dir
• minetest.dir_to_wallmounted
• minetest.wallmounted_to_dir
• minetest.dir_to_yaw
• minetest.yaw_to_dir
• minetest.is_colored_paramtype
• minetest.strip_param2_color
• minetest.player_exists
• minetest.serialize
• minetest.deserialize (?)
• minetest.compress
• minetest.decompress
• minetest.rgba
• minetest.encode_base64
• minetest.decode_base64
• minetest.global_exists

[TheEt1234]

oh wow that's a lot of things, i will simplify the list

things will update as i go and implement the things from the list

Also edit: everything was reworked somewhat, here is how the functions are edited:

• string sandbox gets escaped
• arguments get checked for string length, if it exceeds 64000 it will error
• basic pcall sandboxing is applied (if not debug.gethook() then error("the hook went poof",2))

"+" - will add fully (maybe still needs to be reworked) "+/" - will have to be reworked or limited in a different way "-" - won't be happening "~" - i feel like that's useless

    - ItemStack (why)
    - ItemStackMetaRef (why v2)
    - PerlinNoise  (Due to async limitations i can't)
    - PerlinNoiseMap (same thing)
    - minetest.get_modnames
    +/ minetest.get_game_info (don't expose path, since some people have their real life name as their username for some reason?)
    + minetest.is_singleplayer
    + minetest.features
    ~ minetest.has_feature
    + minetest.get_version
    + minetest.sha1
    + minetest.sha256
    + minetest.colorspec_to_colorstring
    + minetest.colorspec_to_bytes
    + minetest.encode_png
    + minetest.urlencode
    - minetest.string_to_privs (no)
    - minetest.privs_to_string (no)
    + minetest.formspec_escape (could be useful for another project of mine....)
    + minetest.explode_table_event (useful)
    + minetest.explode_textlist_event (useful)
    + minetest.explode_scrollbar_event (useful)
    + minetest.inventorycube
    ~ minetest.dir_to_facedir
    ~ minetest.facedir_to_dir
    ~ minetest.dir_to_fourdir
    ~ minetest.fourdir_to_dir
    ~ minetest.dir_to_wallmounted
    ~ minetest.wallmounted_to_dir
    ~ minetest.dir_to_yaw
    ~ minetest.yaw_to_dir
    ~ minetest.is_colored_paramtype
    ~ minetest.strip_param2_color
    - minetest.player_exists
    + minetest.serialize (i don't know how theese work but i will assume they are just some sort of pcall(table))
    + minetest.deserialize (same thing)
    + minetest.compress
    + minetest.decompress
    + minetest.rgba
    + minetest.encode_base64
    + minetest.decode_base64
    - minetest.global_exists

outside of minetest:

+ pcall = safe.pcall
+ xpcall = safe.xpcall
+ loadstring = safe.get_loadstring(env)
+ bit = table.copy(bit)
+ vector = table.copy(vector)

[SwissalpS]

Some of those would work well as [mooncontroller] libraries. I might c/p some of your code for https://github.com/SwissalpS/minetest_mooncontroller_libraries once you are done ;)

[SwissalpS]

ItemStack() could be interesting to more easily inspect an item's meta. Not sure though if it would work well constructing an ItemStack from info given by a lua- or detector-tube.

[TheEt1234]

ItemStack() could be interesting to more easily inspect an item's meta. Not sure though if it would work well constructing an ItemStack from info given by a lua- or detector-tube.

You would need to receive an item stack with that meta info right? (or maybe its the item string that contains it?) digilines makes sending/receiving userdata impossible

(If luacontrollers could send something hard to serialize through digilines that would probably cause crash bugs on many digilines devices)

Maybe there could be a function to construct a table from an item string but i don't think a full ItemStack is possible because of async limitations

[TheEt1234]

so i just realised how horrid my code was (i was noob with async and didnt think register_async_dofile was worth the effort)

Expect a huge change soon (mostly just to make reading the code easier)

[TheEt1234]

actually no it would be way too much to basically rewrite this

[TheEt1234]

Also, the changes have been commited and are in https://github.com/TheEt1234/async_controller/blob/main/env_plus.lua

keep in mind this is like the only file that's using minetest.register_async_dofile

and that it has not been tested well and is disabled by default and most likely doesn't work, i have only verified it doesn't crash upon loading and that stuff isnt nil, more development will continue tomorrow

[birdlover32767]

well itemstack afaik doesn't change anything in the world, but it sounds better for lua tubes imo

the minetest.facedir_to_dir stuff could be useful in digibuilders, but that's just for a really specific usage on a non-digilines-registered-digilines-device (aka digibuilders are in their own mod)

and if you're going out to add minetest.features why not do minetest.has_feature too?

P.S: remember to do env_plus = nil at the end of the env_plus.lua to not pollute the namespace

_{also wouldnt it be funny if a mod did _G = nil}

[SwissalpS]

here's an item event from a lua-tube (seeder has some metadata set):

{
    pin = {
        name = "red"
    },
    itemstring = "farming_nextgen:seeder 1 51905 \"\\u0001\\u0002return {[\\\"charge\\\"] = 900000}\\u0003\"",
    velocity = {
        speed = 1,
        x = 1,
        y = 0,
        z = 0
    },
    item = {
        count = 1,
        wear = 51905,
        meta = {},
        metadata = "return {[\"charge\"] = 900000}",
        name = "farming_nextgen:seeder"
    },
    type = "item",
    tags = {},
    side = "red"
}

and some comments from my inspection script:

  -- never charged chargeable item
  --t = { metadata = 'return {["charge"] = 0}' }
  -- fully charged/repaired item(s)
  --t = { wear = 1 }
  -- any other charge-/repair-able item(s)
  --t = { wear = { 2, 65536 } }

To detect a never charged or fully depleted seeder, the metadata needs to be inspected. With the seeder this is not too complicated, there are other items that can carry more complex metadata.

[TheEt1234]

well itemstack afaik doesn't change anything in the world, but it sounds better for lua tubes imo

Yeaah and due to minetest limitations i don't think i can create an ItemStack inside the async environment

the minetest.facedir_to_dir stuff could be useful in digibuilders, but that's just for a really specific usage on a non-digilines-registered-digilines-device (aka digibuilders are in their own mod)

I see

and if you're going out to add minetest.features why not do minetest.has_feature too?

Feels useless, you can just do minetest.features[key] == nil or something

P.S: remember to do env_plus = nil at the end of the env_plus.lua to not pollute the namespace

[TheEt1234]

also, because of security issues, minetest.serialize and minetest.deserialize won't be in env_plus anymore

(they use loadstring with bytecode and string.dump which im unsure about)

So instead of that, i've exposed dump and dump2 which can be used to generate valid lua code so you can do pcall(loadstring(dump(table)) (but doesn't allow storing functions)

[birdlover32767]

ok i totally forgot and didnt check that env_plus was local and that itemstacks can be in a table

[TheEt1234]

does anyone want to volunteer to test my horrible code

[birdlover32767]

also my_code should be a thing e.g you can do print(my_code) and the result would be (load):1: print(my_code)

btw what is this error message

[TheEt1234]

also my_code should be a thing e.g you can do print(my_code) and the result would be (load):1: print(my_code)

sure, that could be possible, i will go implement it

btw what is this error message

see print log, it gives you the stack traceback and the error message, maybe the traceback could be modified to be less spammy, if you have some suggestions on how to do it and not mess up in specific cases let me know

and i also need to know how to do it since im not good in lua string manipulation lol

[TheEt1234]

Done in the latest release you can do print(code) if you have env_plus enabled [it's a setting]

[birdlover32767]

yea traceback should be less spammy, maybe shorten the worldpath to just async_controller/sandbox.lua

[TheEt1234]

yea traceback should be less spammy, maybe shorten the worldpath to just async_controller/sandbox.lua

yeah ok but how

and i'd also love to strip out the sandboxing garbage that appears in every traceback (in xpcall: ....)

[birdlover32767]

limit the worldpath to like, the last 15-7 characters

[S-S-X]

Worldpath isn't really useful for players, additionally internal stuff in traceback is also useless for players. Line numbers would be a lot more useful if there would also be line numbers in editor, however not sure if you can have line numbers in reliable way. Probably can't.

[TheEt1234]

limit the worldpath to like, the last 15-7 characters

Yeah ok if i had a reliable way to detect the worldpath i absolutely would love to remove it

Line numbers would be a lot more useful if there would also be line numbers in editor, however not sure if you can have line numbers in reliable way. Probably can't.

I assume every sane person that is programming with luacontrollers is using an external IDE (i personally use vscode.dev)

This is also the reason why i decided to include the traceback in the first place

but also, i have no idea how to strip the internal stuff (sandboxing garbage) and the worldpath from the traceback maybe i should just make my own traceback with debug.getinfo?

well i think i will have to do just that... (doesn't seem too difficult tbh, lua itself has a tutorial on it https://www.lua.org/pil/23.1.html )

[S-S-X]

I assume every sane person that is programming with luacontrollers is using an external IDE (i personally use vscode.dev)

They wont and it is harder than just writing code in game.

Yes I know you will find that statement very weird, however I'm not talking about people who know at least something about programming. I am talking about average players who have no idea, never seen IDE, all they've seen is few code examples that they might write down to post it note.

Adapting technology and mindset to something you're not used to is hard, it has nothing to do with what is efficient after you know at least a bit about what you're doing. IDE (and learning how to use weird very unfamiliar editor) comes after you learn what programming actually means but Minetest on the other hand can teach you basics before you know anything about it.

[TheEt1234]

They wont and it is harder than just writing code in game.

Yes I know you will find that statement very weird, however I'm not talking about people who know at least something about programming. I am talking about average players who have no idea, never seen IDE, all they've seen is few code examples that they might write down to post it note.

Adapting technology and mindset to something you're not used to is hard, it has nothing to do with what is efficient after you know at least a bit about what you're doing. IDE (and learning how to use weird very unfamiliar editor) comes after you learn what programming actually means but Minetest on the other hand can teach you basics before you know anything about it.

i see, but surely the function names will be at least somewhat useful for everyone

[TheEt1234]

Something like this?

The code was:

function test(x)
    error("Successfully failed")
end

function x()
    test()
end

x()

[TheEt1234]

But my current approach doesn't really work out when i modify x() to return test() instead

Idk if i would say it is misleading, but it is helpful, as it will lead you to the function x, then you can discover that the return test() is making an error

The code was:

function test(x)
    error("Successfully failed")
end

function x()
    return test()
end

x()

[birdlover32767]

well that's cool do that ig

btw the print section could use some touch-ups

[TheEt1234]

well that's cool do that ig

btw the print section could use some touch-ups

Yeah fair... do i put in the background

Or do i make it a terminal interface like in the mooncontroller?

or should i allow you to put arbitrary or touchscreen-like formspecs in the print section

[TheEt1234]

And should i put in some in-game docs like the mooncontroller did?

[birdlover32767]

sure

btw do the print log like mooncontroller

[TheEt1234]

sure

btw do the print log like mooncontroller

so even the terminal stuff?

[TheEt1234]

oh that was surprizingly easy

[birdlover32767]

cool

[TheEt1234]

Soo uhh, things to do tomorrow i guess: 1) in game documentation and examples, make it friendly towards new users, even new users to programming (basic lua guide :p?) 2) terminal input 3) test the hell out of env_plus 4) https://github.com/minetest-mods/mesecons/issues/415 5) make a PR to mesecons to finally patch this pesky issue https://github.com/minetest-mods/mesecons/issues/516 [a fix like that is implemented inside the async controller]

[TheEt1234]

ok i did not expect github to mention this. across those issues.. why does it do that