OpenFn / lightning

OpenFn/Lightning ⚡️ is the newest version of the OpenFn DPG and provides a web UI to visually manage complex workflow automation projects.
https://openfn.github.io/lightning/
GNU Lesser General Public License v3.0
125 stars 34 forks source link

Variable limits for runs/dataclips (mem, disk, time) by project #2045

Open taylordowns2000 opened 4 months ago

taylordowns2000 commented 4 months ago

Right now, limits are only set at the instance level, but we need to support setting limits at the project level also. This epic includes a front-end view of the limits for a given project, and several backend changes to ensure that limits are being passed properly to the ws-worker on a per run basis (so that we can define them based on their project—and associated plan, if present).

ToDo:

Note: we know that right now, these limits are respected at least at the level the instance through the use of the following environment variables: https://openfn.github.io/lightning/deployment.html#limits

WORKER_MAX_RUN_MEMORY_MB - how much memory (in MB) can a single run use? WORKER_MAX_RUN_DURATION_SECONDS - the maximum duration (in seconds) that workflows are allowed to run (keep this below your termination_grace_period if using kubernetes) WORKER_CAPACITY - the number of runs a ws-worker instance will take on concurrently. MAX_DATACLIP_SIZE_MB - the maximum size (in MB) of a dataclip created via the webhook trigger URL for a job. This limits the max request size via the JSON plug and may (in future) limit the size of dataclips that can be stored as run_results via the websocket connection from a worker.

taylordowns2000 commented 4 months ago

@christad92 , would love to chat with you about this!