search

colcon / colcon-parallel-executor

Extension for colcon to process packages in parallel
http://colcon.readthedocs.io
Apache License 2.0
2 stars 4 forks source link

-j for jobs #24

Closed DLu closed 3 years ago

DLu commented 3 years ago

Both catkin_make and catkin_tools use -j/--jobs to specify the number of parallel workers. This PR adds those to the argparse parser as aliases.

dirk-thomas commented 3 years ago

In catkin_make there is no parallel processing of packages. These arguments are directly passed to make and therefore work exactly as provided by that underlying tool (https://www.gnu.org/software/make/manual/html_node/Parallel.html).

In catkin_tools multiple packages can be processed in parallel but a job server ensures that the desired number of threads isn't exceeded across all of these (https://catkin-tools.readthedocs.io/en/latest/verbs/catkin_build.html#configuring-build-jobs).

In these two cases the arguments provide the same behavior as make so naming then the same makes sense.

But in colcon that is different. It does process packages in parallel but doesn't implement a job server. So the semantic of this parameter is different. Therefore I don't think it should be offered under the same name as the one from make. This will likely confuse users which will have the expectation that an argument with the same name had the same behavior.

tylerjw commented 3 years ago

It does process packages in parallel but doesn't implement a job server.

From the perspective of a user, how is this different? To me, it seems like a semantic difference between calling fork and asking make to call fork. Am I wrong?

Generally from a user-experience perspective, this is just one more thing that is annoying to deal with when the old way was much nicer to the user (interface more closely resembled that of a familiar tool, make).

dirk-thomas commented 3 years ago

From the perspective of a user, how is this different?

A user which knows make will expect that only N threads will be used when passing -j N. But that would not be the behavior this option will give you. Instead it will process as much as N packages in parallel - each with up to M (defaulting to your number of cores - your actual make -j) threads.

tylerjw commented 3 years ago

Is there a way to limit the number of threads used by make to less than number of cores with colcon then (if you don't have enough ram for -j)?

dirk-thomas commented 3 years ago

Is there a way to limit the number of threads used by make to less than number of cores with colcon then?

https://answers.ros.org/question/368249/colcon-build-number-of-threads/

tylerjw commented 3 years ago

Thank you for explaining how it works. This makes much more sense now.