search

slimtoolkit / slim

Slim(toolkit): Don't change anything in your container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)
Apache License 2.0
19.5k stars 732 forks source link

Ability to run http probes concurrently to decrease build time #711

Open tylersalminen opened 2 months ago

tylersalminen commented 2 months ago

The ability to run http probes concurrently (up to a defined number of requests) could greatly decrease the build time -- especially when probing IO bound features of an application in an image. It appears that only the http probe crawler feature has such a concurrency feature.

kcq commented 2 months ago

@tylersalminen what kind of application do you have? What's your tech stack? What's IO bound in your application? It should be possible to have a basic version of it, --http-probe-concurrency, to get started. Later the custom problem will also need concurrency groups to make sure certain calls happen after certain conditions happen or after certain calls (e.g., a probe that deletes a record should happen after a call that creates that record).

tylersalminen commented 2 months ago

The application is an in house built web app / platform for hosting custom finance applications at the company I work for. So pretty much most of the applications functionality is io bound (ie forwarding network calls to various databases and services and maybe a little processing on the web app to provide some functionality our underlying services do not).

One particular set of probes I currently do to build our image involves probing our automation / jobs feature that allows users to export pages / reports to pdf and excel with triggers / schedules. This utilizes a large number of libraries as well as an actual headless chrome browser to provide the report generation. Probing these features are particularly io bound, ie each individual http call to these features can wait for lots of serially dependent network calls to our backend services in order to generate the report.

My particular probes are not dependent on each other so establishing dependencies between them is not a problem for my particular use case but I do understand your point.

kcq commented 2 months ago

Thank you for the context @tylersalminen ! Very useful!

What language do you use in your app? JavaScript/TypeScript/node? Python? Which web/api framework do you use?

Is it an SPA or similar app (e.g., Next.js)? How dynamic is your web app? Do you have a lot of static resource and/or templates your browser side app loads at runtime?

P.S. --http-probe-concurrency is on its way... and it'll be in the next release :-)