Add async/await as better way to call blocking functions

[CrosRoad95]

Is your feature request related to a problem? Please describe. Callbacks hell, its often hard and uncomfortable to pass all data you need to next function and some of them just can't be passed

Describe the solution you'd like add build in mta async/await style functionality, eg.:

invokeAsync(function()
  response = fetchRemoteAsync("mtasa.com/api")
  data = fromJSONAsync(response.data)
  print(data)
  dbQueryAsync("insert into...", data)

  data = triggerClientEventAsync("getFoo", getRandomPlayer())
  print("data", data) -- prints "foo"

  while true do -- equivalent of setTimer(print, 100, 0, "hi")
    sleep(100)
    print("hi")
  end
end)

-- clientside:
addEventHandlerAsync("getFoo", root, function()
  return "foo"
end)

less code, more readable, easy to use, non blocking

Describe alternatives you've considered /

Additional context https://forum.mtasa.com/topic/54787-simple-sleep-function/ example how to do this

Related issues #1815

[Pirulax]

For this MTA would need to support MT properly, and it doesn't. Also Lua doesn't either, although this can be solved by adding GIL, which would slow down single threaded scripts a little, but nothing horrible, MTA is the blocker here.

Also, this approach isn't the greatest. Python's is way better, as we wouldn't have duplicate functions, which can be confusing, especially to new scripters.

[CrosRoad95]

Someone, most experienced person should make an example, template of how to make async, multithreaded function and then everyone else should follow the same rules

yes, i know 2 functions, one sync, second async may be confusing. Solution for that would be api versioning, but idk

[Pirulax]

The biggest problem here isn't the function naming. It's that MTA doesn't support multi threading by default. If you were to call any of the function's from a different thread, there's a high chance of it crashing. So unless we reprogram the whole MTA to be async this won't work. And even then GTA isn't multi threaded either, so we'd still have a bottleneck somewhere. Those few functions that can be run totally async already have the callback option.

[4O4]

For this MTA would need to support MT properly, and it doesn't. Also Lua doesn't either, although this can be solved by adding GIL, which would slow down single threaded scripts a little, but nothing horrible, MTA is the blocker here.

I don't like to say people they are wrong, but you are wrong on this one mate :) You actually don't need multi threading for async/await at all. Javascript is a single threaded environment running an event-loop with occasional APIs which are running in separate threads (like IO), and there is a great async / await support there. This is very similar to how MTA works. We have a main game loop, a lot of events and also some APIs which are processed in a separate threads (like fetchRemote, dbQuery, triggerLatent...). What's worth pointing out is that the coroutines in Lua are basically the exact equivalent of async / await functionality in JS. Both a/a and coroutines allow you to write an async code that is flat like a synchronous code (no nested callbacks hell, if you don't know what callback hell is then you are one lucky and happy man 😄 See: https://callbackhell.com ).

This issue by @CrosRoad95 is actually related to what I was talking about yesterday on the #development channel and I wanted to publish a similar proposal about adding a first class support for coroutines in the existing asynchronous APIs in the MTA. By first class support I mean the functions that yield the currently running thread, and this means that these functions could only be invoked from the coroutine threads (not actual real OS-level threads, it's all about plain Lua coroutines!).

To get some broader picture of that please take a Roblox game as an example, look at their docs: https://developer.roblox.com/en-us/api-reference/function/HttpService You will notice that some function names are ending with Async word and are marked with [yields], i.e. GetAsync, PostAsync. Here's a full signature from the docs:

string GetAsync ( string url , bool nocache , Variant headers ) [Yields] 

AFAIK the Roblox environment is a bit different because everything there is based on coroutines and they actually have a way of spawning OS-level threads, but as I said earlier this is completely not needed. You can combine the ideas and have the best of both worlds. The existing MTA Lua APIs which already take callbacks as arguments could be used this way. Of course you can do wrappers on the Lua level to make it work the same way, but I'll repeat what I said earlier - I suggest adding a first class support for that. This can contribute to the overall code quality on the Lua side and will help the developers to avoid callback hell if / when more async APIs are implemented especially around IO and other heavy operations.

Another Lua platform worth taking a closer look at is Lusty web framework running on OpenResty. Quoting from their website (https://leafo.net/lapis/):

Nginx’s event loop is used for all asynchronous actions, including HTTP requests and database queries. With the power of Lua coroutines code is written synchronously but runs asynchronously, without all that callback spaghetti seen in other asynchronous platforms. It’s fast, easy to read, and easy to write.

Here's also an excellent blog post about that: https://leafo.net/posts/itchio-and-coroutines.html

So to summarize: Async/await is not tied to OS-level multithreading support. Async/await suits well in every environment which is driven by events, and MTA is for sure such environment. Lua coroutines are basically a drop-in replacement for async/await functionality as known from other languages such as Javascript. The proposal is to bring a first class support for coroutine based async APIs which would improve the overall development experience and would help avoid spaghetti code when a lot of async code is in play on the Lua level. There are existing codebases that we can take inspiration from,

[qaisjp]

Implementing coroutine-versions of the MTA API should already be possible, in plain Lua. Please can someone build a proof of concept?

[CrosRoad95]

can you provide details how it should work? i could try do this

[qaisjp]

--
-- async.lua (library)
--

-- async returns a version of fn that has MTA functions running in an async context
function async(fn)
    -- to be implemented by CrosRoad95
end

--
-- client.lua (user code)
--

function hash()
    local str = passwordHash("test", "bcrypt", { cost = 20 })
    outputChatBox(str)
end

addEventHandler("onClientResourceStart", resourceRoot, async(hash))

This should run the hash asynchronously without locking up the client.

[CrosRoad95]

do this need to be in c++, or plain lua the same as inspect, exports ?

[Pirulax]

Implementing coroutine-versions of the MTA API should already be possible, in plain Lua.

[CrosRoad95]

yep, it is possible

_passwordHash = passwordHash
function passwordHashAsync(...)
    local args = {...}

    local co = coroutine.running() 
    local result;
    local function resumeThisCoroutine(...)
        result = {...}
        coroutine.resume(co)
    end
    _passwordHash(args[1], args[2], args[3], resumeThisCoroutine)
    coroutine.yield()
    return unpack(result)
end

function async(fn)
    local co = coroutine.create(fn)
    coroutine.resume(co)
    return function()end
end

function await(func, ...) 
    local co = coroutine.running() 
    local function resumeThisCoroutine(arg)
        coroutine.resume(co)
    end 
    setTimer(resumeThisCoroutine, 100, 1)
    coroutine.yield()
    return "Asdf"
end 

function hash()
    local str = passwordHashAsync("test", "bcrypt", { cost = 15 })
    iprint("result",str)
end

addEventHandler("onResourceStart", resourceRoot, async(hash))

[Pirulax]

That seems very over-complicated just to not use a callback directly.

[qaisjp]

It looks like your script runs hash immediately, and does not return an async version of the function. Try a command handler, addCommandHandler("test", async(hash)).

You should use setfenv to modify the function environment, i.e. to translate passwordHash to an async version in just that function's environment.

[4O4]

It looks like your script runs hash immediately, and does not return an async version of the function. Try a command handler, addCommandHandler("test", async(hash)).

You should use setfenv to modify the function environment, i.e. to translate passwordHash to an async version in just that function's environment.

Wouldn't it be better to have a clear visual distinction between sync and async functions though? The existing code needs to be modified anyway to get rid of the callback so it's not like it's going to magically convert someone's callback based code to coroutine based one with just one call of async(). Can you elaborate a little bit more what are your further intentions with the PoC? Also, are there any cons of implementing it on the c++ level? Would be great to get some insights.

In the meantime I've edited the sample from @CrosRoad95 , fixed version below:

local _passwordHash = passwordHash
function passwordHashAsync(...)
    local args = {...}

    local co = coroutine.running() 
    local function resumeThisCoroutine(...)
        coroutine.resume(co, ...)
    end
    _passwordHash(args[1], args[2], args[3], resumeThisCoroutine)

    return coroutine.yield()
end

local asyncEnv = setmetatable({ passwordHash = passwordHashAsync }, { __index = _G })

function async(fn)
  return function() 
    setfenv(fn, asyncEnv)
    local co = coroutine.create(fn)
    coroutine.resume(co)
  end
end

function hash()
  iprint("hash started")
    local str = passwordHash("test", "bcrypt", { cost = 15 })
    iprint("result",str)
end

addEventHandler("onResourceStart", resourceRoot, async(hash))
addCommandHandler("testasync", async(hash))

BTW Keep in mind that test command is reserved internally by MTA I believe, so if you make a command handler for that it will not work. I changed the command to testasync in this PoC.

[Pirulax]

We need to keep backwards compatibility, so we can't require the current code to be changed.

[4O4]

We need to keep backwards compatibility, so we can't require the current code to be changed.

Sorry I don't follow By adding a new function let's say passwordHashAsync you avoid both parameter-overload-related headaches and compatibility issues.

[Pirulax]

Ah, sorry, must have misunderstood / misread something, my bad. Yeah I'm fine with adding a new function to make things more clear.

[CrosRoad95]

better example https://pastebin.com/unrJQ2Jd

for x=1, 3 do
    invokeAsync(function()
        for y=1,3 do
            print("xy",x,y)
            sleep(1000)
        end
    end)
end

needs 3 seconds to finish

[qaisjp]

Can we extend the PoC to support a generic implementation?

local callbacks = {
    passwordHash = 4 -- the callback is at argument 4
}

Wouldn't it be better to have a clear visual distinction between sync and async functions though?

Perhaps an await metatable, dynamically added into the function environment using setfenv?

function printHi()
    local s = await:passwordHash("hi", "bcrypt", { rounds = 20 }
    outputChatBox(s)
end

addCommandHandler("hi", async(printHi))

Can you elaborate a little bit more what are your further intentions with the PoC?

My further intentions of the PoC is that, if this can be done entirely in Lua, someone can publish this and maintain their own package. The semantics of coroutines and this sort of asynchronous behaviour is uncommon in MTA, so we need to be careful.

I'm not entirely against integrating it in MTA, as long as we maybe embed this metadata into the new parser, instead of maintaining another function list. OOP was implemented in a very non-DRY way. We should try and avoid the issue we have with OOP. Including some of the context switching performance issues.

[4O4]

Thanks for the answer. I can understand this point of view:

My further intentions of the PoC is that, if this can be done entirely in Lua, someone can publish this and maintain their own resource. The semantics of coroutines and this sort of asynchronous behaviour is uncommon in MTA, so we need to be careful.

but the problem is that you can't really make a nice utility resource from that because metatables, functions, fenv... it obviously won't cross the resource boundaries when doing inter-resource calls. One way around it would be to embed these utilities in each resource you want to use it in, but then in order to clean the code, you have to pollute the resources first with duplicated files. Other approach involves loadstring hacks to load the script contents from another resource which isn't too pretty either. Third option is require() which we don't have. Also I strongly believe that if the feature would be an official one, properly documented and made clearly visible on wiki pages with official APIs, then it would gain traction more easily than any utility resource maintained by third-party devs.

I'm not entirely against integrating it in MTA, as long as we maybe embed this metadata into the new parser

I've heard from Pirulax about some automagical arguments parser, is this what you mean? I have absolutely no idea how it works, what limitations it has etc. so I won't be very helpful with that now.

Perhaps an await metatable, dynamically added into the function environment using setfenv?

Well... If we're going that way, it could be even made syntactically identical to JS / C#:

local s = await passwordHash("hi", "bcrypt", { rounds = 20 })

or optionally a more visible "code block"-alike variant

local s = await { 
  passwordHash("hi", "bcrypt", { rounds = 20 })
}

I was proposing this variant:

local s = passwordHashAsync("hi", "bcrypt", { rounds = 20 }) -- ...Async sufix = yields

because it could potentially involve less or none of metatables and setfenv magic. It is also easier to document and easier to search by full keyword in the documentation. Wiki pages for functions with lot's of different combinations of optional params / overload signatures can be very messy.

[qaisjp]

but the problem is that you can't really make a nice utility resource from that because metatables, functions, fenv... it obviously won't cross the resource boundaries when doing inter-resource calls.

The Lua packages project #680 aims to resolve that. I think this is a core problem we should solve, instead of working around the problem by integrating everything directly into MTA.

Also I strongly believe that if the feature would be an official one, properly documented and made clearly visible on wiki pages with official APIs, then it would gain traction more easily than any utility resource maintained by third-party devs.

You raise a good point, but also, having things directly into MTA makes it harder to make improvements in the future. There's a much higher bar and backwards compatibility is stricter.

We can always have official MTA libraries with a different "contract" and weaker guarantees for stability / BC.

I've heard from Pirulax about some automagical arguments parser, is this what you mean?

Yes

it could be even made syntactically identical to JS / C#:

This is syntactically incompatible with Lua, as I'm sure you're aware

It is also easier to document and easier to search by full keyword in the documentation. Wiki pages for functions with lot's of different combinations of optional params / overload signatures can be very messy.

Documentation is hard and having to document duplicate pages for everything sounds like a pain, and I think it will look very ugly. Maintaining OOP hints for things is already a pain.

I think the idea here is to treat the code as if it had no callback, but under the hood, it uses callbacks.

A scripter has two options for their mental model:

1. young scripters will decide that the await:F() calls the non-callback versions of F (and does magical async/await things)
2. seasoned scripters will decide that the await:F() places the callback in the right place (and the callback handles magical coroutine stuff) - this is the more accurate version

Are there any functions that support multiple callbacks?

[4O4]

Okay, I think it's now clear what are the overall intentions and constraints, thanks.

AFAIK there are no functions with multiple callbacks. There might be multiple overloads but ultimately it's either 1 or 0 callbacks.

This is syntactically incompatible with Lua, as I'm sure you're aware

Emmm... All examples I posted above are valid Lua syntax which I'm even using right now in my own code :wink: Functions taking one argument which is either a table or a function don't need parentheses to perform invocation. That's what makes it work.

[qaisjp]

Oh, cool, I didn't know that functions could take other functions without spaces! Very nice.

For the table example, how does passwordHash know it's running in the async context? It looks like it just passed a table with size 1, so I assume async is setting a global variable somewhere.

[4O4]

Edit: As it was discussed today on Discord, I was wrong on the JS-like await syntax. You can't have that in plain Lua the way I was describing . Sorry for bringing some false information here. I'm scratching out the parts which are invalid because of that, only leaving the relevant and working parts. Although this syntax can theoretically be achieved in more hacky way as @qaisjp suggested: "It would only work if we added a metatable to _G that set some sort of global flag when you try to access await, and unset the flag when you call a certain function. "

Oh, cool, I didn't know that functions could take other functions without spaces! Very nice.

Without parens, not spaces :) Also I might have oversimplified this because my brain wasn't fully awake yet. What I actually meant was not functions but callable tables, so there is further metatables trickery needed. But the point was that it's doable, not that it's straightforward to do :smile:

For the table example, how does passwordHash know it's running in the async context? It looks like it just passed a table with size 1, so I assume async is setting a global variable somewhere.

There's actually not much magic going on here. It just takes the first element of the table like you said and from there it works the same way as if it was called directly with the argument at index [1]. I prefer this variant personally. This is how DSLs are often made with Lua.

Click to show old and wrong code (for sake of clean history of discussion)

```lua -- This is PoC, don't use for serious stuff without reviewing it first -- Callable table is only needed for JS / C# like syntax without parens and curly braces. -- For codeblock-alike `async { function () end }` and `await { fn() }`, plain functions will do. local _passwordHash = passwordHash local passwordHashAsync = setmetatable({}, { __call = function (...) local args = {...} local co = coroutine.running() local function resumeThisCoroutine(...) coroutine.resume(co, ...) end _passwordHash(args[1], args[2], args[3], resumeThisCoroutine) return coroutine.yield() end }) -- tableOrFn must be either a callable table, or -- a table with a callable table at index [1]. local function await(tableOrFn) local fn = tableOrFn if tableOrFn[1] then fn = tableOrFn[1] end return fn() end local asyncEnv = setmetatable( { passwordHash = passwordHashAsync, await = await, }, { __index = _G } ) function async(fn) local fn = fn if type(fn) == "table" then fn = fn[1] end return function (...) setfenv(fn, asyncEnv) local co = coroutine.create(fn) coroutine.resume(co, ...) end end ``` Usage: ```lua function hash() iprint("hash started") local str = await passwordHash("test", "bcrypt", { cost = 15 }) --[[ OR local str = await { passwordHash("test", "bcrypt", { cost = 15 } } ]] iprint("result", str) end addEventHandler("onResourceStart", resourceRoot, async(hash)) addCommandHandler("testasync", async(hash)) ``` Alternative syntax: ```lua hash = async { function (...) iprint("hash started", ...) local str = await passwordHash("test", "bcrypt", { cost = 15 }) iprint("result", str) end } addEventHandler("onResourceStart", resourceRoot, hash) -- hash is an already "asyncfied" function addCommandHandler("testasync", hash) ``` Writing by hand from memory, not really tested but should work

-- This is PoC, don't use for serious stuff without reviewing it first
local _passwordHash = passwordHash
local function passwordHashAsync(...)
    local args = {...}

    local co = coroutine.running() 
    local function resumeThisCoroutine(...)
        coroutine.resume(co, ...)
    end
    _passwordHash(args[1], args[2], args[3], resumeThisCoroutine)

    return coroutine.yield()
  end

-- tableOrFn must be either a function, or
-- a table with a function at index [1].
local function await(tableOrFn)
  local fn = tableOrFn

  if tableOrFn[1] then
    fn = tableOrFn[1]
  end

  return fn()
end

local asyncEnv = setmetatable(
  { 
    passwordHash = passwordHashAsync,
    await = await,
  }, 
  { __index = _G }
)

function async(fn)
  local fn = fn

  if type(fn) == "table" then
    fn = fn[1]
  end

  return function (...) 
    setfenv(fn, asyncEnv)
    local co = coroutine.create(fn)
    coroutine.resume(co, ...)
  end
end

Usage:

function hash()
  iprint("hash started")
  local str = await(passwordHash("test", "bcrypt", { cost = 15 }))

  --[[ 
  OR

  local str = await {
    passwordHash("test", "bcrypt", { cost = 15 }
  }
  ]]

  iprint("result", str)
end

addEventHandler("onResourceStart", resourceRoot, async(hash))
addCommandHandler("testasync", async(hash))

Alternative syntax:

hash = async {
  function (...)
    iprint("hash started", ...)
    local str = await {
      passwordHash("test", "bcrypt", { cost = 15 })
    }
    iprint("result", str)
  end
}

addEventHandler("onResourceStart", resourceRoot, hash) -- hash is an already "asyncfied" function
addCommandHandler("testasync", hash)

[prnxdev]

2 propositions:

1. Add proper async/await functionality to MTA
2. Add JS's Promise-like notation but only when OOP is enabled. If it's not then we can use callbacks

   fetchRemote('https://some.api/user')
   .then(function(response) ... end)
   .catch(function(error) ... end)

   and implement it to every possible functions like fetchRemote, fileOpen etc.

[CrosRoad95]

1 yes, 2 NON ONO NO NO NO NO

[4O4]

I don't see any benefits from bringing in the promise like interfaces. Promises are JS thing and whole async await in JS is built around them. In C# async await is built around Tasks. In Lua, async await is kind of already built in in the form of coroutines (when it comes to control flow behavior, not syntactically). Promise like interfaces sound to me like an artificial noise.

I don't see anything wrong in implementing and using them individually in own resources or as additional lua lib, but I just don't feel like this is needed as a core building block for anything related to this async await proposal.