search

crystal-lang / shards

Dependency manager for the Crystal language
Other
758 stars 99 forks source link

Allow building multiple build targets in parallel #545

Closed carlhoerberg closed 8 months ago

carlhoerberg commented 2 years ago

Pass -j X or --jobs=X to start X build process in parallel to speed up building applications with multiple targets (with the cost of more memory).

Blacksmoke16 commented 2 years ago

Might be better to contribute to the discussion/progress of https://github.com/crystal-lang/shards/pull/539 instead of introducing a new approach?

asterite commented 2 years ago

I was thinking about that... but I think this is about building things in parallel, while the other one is fetching things in parallel... which of course will be very confusing if we have two flags to control these.

Blacksmoke16 commented 2 years ago

Oh sorry, I didn't notice this was for building. But yea, that's a good point. Maybe the flag could be consolidated/generalized to support both cases?

carlhoerberg commented 2 years ago

Maybe, but having a default > 1 might cause memory issues as each build process uses quite a lot of memory. Hence you might want different defaults for --jobs and --parallel-fetch..

straight-shoota commented 2 years ago

I don't like this because I don't want shards to become more of a build tool. It should focus on managing dependencies, and IMO does not need advanced build features.

And there are many alternatives. If you use an actual build system (which I would recommend for any project), it probably already has an option for parallel building (make -j for example). This is also easy to do with the parallel command: parallel shards build ::: target1 target2.

m-o-e commented 2 years ago

might cause memory issues as each build process uses quite a lot of memory Hence you might want different defaults for --jobs and --parallel-fetch

Agree. If both get added they definitely need to have two separate flags.

In bundler --jobs means: The maximum number of parallel download and install jobs. (man-page) In make --jobs means: Specifies the number of jobs (commands) to run simultaneously. (man-page)

Our own --jobs flag can only be compatible with one of them, not both at the same time. (as the two tend to have different optimal values, esp. on low-memory machines)

Since shards is conceptually closer to bundler than make, I would vote for adopting the bundler meaning of --jobs (as already done in #539).

If this one should also be added then I'd suggest to call the flag here --parallel-build (or similar).

I don't want shards to become more of a build tool.

I'd be inclined to agree with this as well. The default for this will probably always have to be 1. Meaning anyone who wants to use it will have to change their invocation.

And at that point they could as well replace it with something involving parallel.

I suppose the argument for having this functionality baked in could be to make it more discoverable/easier, tho. 🤔

hugopl commented 8 months ago

This is that kind of feature that at first looks very nice, but then you notice that we are crossing the borders between a dependency manager and a building tool... years later you notice that you just wrote a bad version of make.