Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal.
Not entirely sure how this should work or if it's a use case you want to support.
Problem:
I have a long-running script that takes 10 files as input; most of the time is spent waiting for network requests
the first step of that script is to run OCR on each file: send the file to a OCR service and wait until it's processed, then return the text; this takes e.g. 10 minutes
the OCR is in a separate script
I'm currently running everything in a main python script that imports the OCR script and asyncio.gather s the 10 files
this works fine, but:
the first 10 minutes of runtime are spent waiting for the api calls to return, and the worker is blocked during that time even though it's just waiting on API calls
I can't run the ocr script as a separate job, with wmill.run_script, which would e.g. let me take advantage of built-in cache features, without tying up an extra worker per file for 10 minutes
Proposal:
windmill scripts with async def main could run concurrently on one worker. The worker would run the eventloop.
Not entirely sure how this should work or if it's a use case you want to support.
Problem:
wmill.run_script
, which would e.g. let me take advantage of built-in cache features, without tying up an extra worker per file for 10 minutesProposal: