search

windmill-labs / windmill

Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool.
https://windmill.dev
Other
9.6k stars 438 forks source link

feature: enable Deno KV for Deno #4145

Open thoughtpolice opened 1 month ago

thoughtpolice commented 1 month ago

Deno KV is a key-value store API for Deno that will be going stable in 2.0. I'd like it if the API was available to my Windmill scripts.

Normally, KV wouldn't make much sense for Windmill because Deno.openKv() by itself just opens a local (sqlite) database on the filesystem and such local state doesn't have a clear lifetime. But a very neat feature is that you can also use Deno KV with remote servers, and therefore with multiple Deno processes, by calling openKv("https://kv.example.com:4512"), where that URL points to either Deno Cloud or a locally running denokv service that you host yourself.

My motivation is that I have a long-running service[1] written in Deno that does not use Windmill, but I want to have some Windmill scripts and flows that it works in tandem with. Therefore, I'd like my service and my (Deno) Windmill scripts to both use openKv with a shared server for shared state; for example a Windmill script might be invoked and write some data to the database that the outside service can read later.

This feature can be enabled with the --unstable-kv flag passed to the runtime. Based on https://github.com/windmill-labs/windmill/commit/8d5c7c2b461ee71b8376b12f574dffff1c9f1387 (found via blame) which added a similar flag, it seems pretty easy to fix this, if a patch would be acceptable.

[1] In theory it could use Windmill, it's a pure Deno script, but for various reasons I'd really like it to be a persistently running service which has a clear persistent lifetime (e.g. the typical cold start concerns). What I really want it something like deno serve as a first-class concept in Windmill, but I realize that's a different discussion entirely...

thoughtpolice commented 1 month ago

Oh, I forgot that you also need to set the DENO_KV_ACCESS_TOKEN environment variable in the context of deno for this to work, which is the access token for the endpoint. That should work well enough with contextual variables, I think. It would be more ideal if you could have per script/flow/app contextual variables instead of just per-workspace variables, but I can manage for now.

rubenfiszel commented 1 month ago

It's a great idea, I suggest the flag is passed if the env variable is available to the worker. To make this slightly faster, it should be cached in memory with a lazy static

Le dim. 28 juil. 2024, 20:17, Austin Seipp @.***> a écrit :

Deno KV is a key-value store API for Deno that will be going stable in 2.0. I'd like it if the API was available to my Windmill scripts.

Normally, KV wouldn't make much sense for Windmill because Deno.openKv() by itself just opens a local (sqlite) database on the filesystem and such local state doesn't have a clear lifetime. But a very neat feature is that you can also use Deno KV with remote servers, and therefore with multiple Deno processes, by calling openKv("https://kv.example.com:4512"), where that URL points to either Deno Cloud or a locally running denokv https://github.com/denoland/denokv service that you host yourself.

My motivation is that I have a long-running service[1] written in Deno that does not use Windmill, but I want to have some Windmill scripts and flows that it works in tandem with. Therefore, I'd like my service and my (Deno) Windmill scripts to both use openKv with a shared server for shared state; for example a Windmill script might be invoked and write some data to the database that the outside service can read later.

This feature can be enabled with the --unstable-kv flag passed to the runtime. Based on 8d5c7c2 https://github.com/windmill-labs/windmill/commit/8d5c7c2b461ee71b8376b12f574dffff1c9f1387 (found via blame) which added a similar flag, it seems pretty easy to fix this, if a patch would be acceptable.

[1] In theory it could use Windmill, it's a pure Deno script, but for various reasons I'd really like it to be a persistently running service which has a clear persistent lifetime. What I really want it something like deno serve https://docs.deno.com/runtime/manual/tools/serve/ as a first-class concept in Windmill, but I realize that's a different discussion entirely...

— Reply to this email directly, view it on GitHub https://github.com/windmill-labs/windmill/issues/4145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDJAAKU6ZIQWMUSRU3VLDZOUYULAVCNFSM6AAAAABLTABXSOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQZTIMBVGM2DIOA . You are receiving this because you were assigned.Message ID: @.***>