choria-io / asyncjobs

Asynchronous Job Processor
Apache License 2.0
89 stars 5 forks source link

Add wasm based tasks #117

Open gedw99 opened 1 year ago

gedw99 commented 1 year ago

Would allow nats obj to store the wasm and then run the wasm.

why ?

https://github.com/bots-garden/capsule Is a pure gol g wasm runner. So a decent candidate . Has no db dependencies. Used to work with nats butbis being refactored to have more formal API encapsulation.

@ripienaar if you have time let me know what you think

there are other Wazero based runners. Capsule is just one

ripienaar commented 1 year ago

For the next month or 2 I am on sabatical so no time soon - but this is an area close to my heart so would be keen to think it through.

Immediate question would be where would you deploy the wasm workers - would still need to be some kind of fat host, probably a container, so wonder what would really end up being the big win in terms of deployment.

gedw99 commented 1 year ago

hey @ripienaar

Holidays are good :) me too this summer. Going to spain.

Let’s support both wasm and docker .?

For me the lightness of Wazero and no cgo is useful for the open source science project I am working on. I feel other will also have a need edge jobs too where docker is too heavy. Nats leaf etc :)

https://github.com/stealthrocket/wasi-go Is the runner that uses Wazero. Works currently without tinygo if you use gotip. Can only run golang wasm but they are planning on python, js, .net, etc.

We probably need to start with an agent that uses the nats / async api directly ? One agent per machine.

If i took wasi-Go as a basis, then the agent would need to control wasi-go.

The same for docker. The agent would need to spawn docker containers.

I already have some of this rigged up . A gui that talks to async. An agent that talks to async.

Let me know your thoughts on my concept :)