Pg executor - Githubissues

insitro / redun

Yet another redundant workflow engine

https://insitro.github.io/redun/

Apache License 2.0

510 stars 43 forks source link

Pg executor #84

Open gabriel-v opened 12 months ago

gabriel-v commented 12 months ago

closes https://github.com/insitro/redun/issues/78

If you want to accept this, I guess this would be the minimum left to do

[x] add CLI subcommand to start executor worker process
[x] add&fix type annotations
[ ] add instructions & docs sections for running
[ ] test internals of executor (send/receive to/from queue)
[x] test a bunch of different / missing settings in the ini file
[x] test executor with workflows

gregjohnso commented 7 months ago

@mattrasmus this would be very helpful for us. What is the approval mechanism here?

gabriel-v commented 4 months ago

hi @ernestum

Thanks for the review. I'm no longer working on this experiment, it was a nice starting point into understanding how this works; I ended up working on a new framework using different opinions, requirements and backends

Feel free to take over the code if you want to further pursue this feature

ernestum commented 4 months ago

All right. Will do. Thanks for the huge initial chunk of work!

May I ask what other framework you ended up using @gabriel-v?

gabriel-v commented 4 months ago

Did a custom reinterpretation of this repo's DAG definition API using async python and scylladb, seeing some great throughput out of it - will leave a comment here when we publish it if you're interested

ernestum commented 4 months ago

Did a custom reinterpretation of this repo's DAG definition API using async python and scylladb, seeing some great throughput out of it - will leave a comment here when we publish it if you're interested

Yes that would be great!

gabriel-v commented 3 months ago

meanwhile you can look at https://github.com/temporalio/samples-python

we're evaluating it and it's a very strong contender