op.SetRunParallel(True) causes issues with multiprocessing

[bragostin]

In shapes.py the _bool_op() function is set with op.SetRunParallel(True) by default.

I am using the optimization library rbfopt to perform a shape optimization. the CQ object is quite small (500 ko), but with 4 geometric parameters to optimize it usually takes 300 iterations. Furthermore on top of that there is a list of configurations to optimize. Therefore I am using the multiprocessing option of rbfopt to accelerate the execution because the CQ object is small but the optimization is complex.

However op.SetRunParallel(True) prevents me from using the multiprocessing option of rbfopt. After the first optimization of the list is done the process hangs indefinitely. But if I set op.SetRunParallel(False) it works as intended.

Would there be a way to force op.SetRunParallel(False) when needed?

[bragostin]

For now I have found a workaround by patching shapes.py, launching cq-editor and unpatching like this:

FILE=$(find ~ -path /*/envs/cadquery/lib/*/site-packages/cadquery/occ_impl/* -name shapes.py)
sed -i 's/op.SetRunParallel(True)/op.SetRunParallel(False)/g' $FILE
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia cq-editor
sed -i 's/op.SetRunParallel(False)/op.SetRunParallel(True)/g' $FILE

Do you think it would make sense to expose the SetRunParallel option in fuse, cut, split and intersect? It would seem that having multiprocessing enabled by default can cause interactions with other libraries. If yes, would you accept a PR?

[adam-urbanczyk]

Are you referring to cq.Shape methods? If so, I guess yes. If it is more general, maybe we need to think about some kind of configuration management framework (e.g. gin-config or hydra).

[bragostin]

@adam-urbanczyk yes I was referring to 'cq.Shape' methods. But there are quite a few cq methods using these, all would need to have SetRunParallel exposed.

[adam-urbanczyk]

That was indeed my point regarding configuration management - I don't think it is good to expose SetRunParallel on every level. Even on the cq.Shape level it might be a good idea to use some kind of global configuration.

[bragostin]

@adam-urbanczyk thank you, I will have a look at gin-config and hydra, if it allows to change SetRunParallel at the script startup then it sounds like a good solution because I guess this option will rarely need to be changed.

[adam-urbanczyk]

Just to be clear, AFAIK you'd still need to add this option to cq.Shape.

[bragostin]

@adam-urbanczyk Thing is, I need to put back the SetRunParallel option to True at a later stage (basic shape is optimized according to some physical law and then patterned into a much bigger part), and it does not seem gin-config or hydra would allow that, from what I read. When I use cq-editor I tried to use importlib.reload(cq) after modifying shapes.py but it does not have any effect. It would be an easy solution though.

[adam-urbanczyk]

Why at a later stage and not at the beginning?

[bragostin]

@adam-urbanczyk There are two stages in the script:

• optimization of a basic cq object according to fluid dynamics laws. SetRunParallel needs to be set to False otherwise the optimizer cannot use multiprocessing.
• building of the final object with patterning of the optimized basic shape + some Boolean operations. SetRunParallel would need to be set back to True

[adam-urbanczyk]

OK, somewhat clearer. I think this should be supported by the frameworks mentioned. Worst case, you can modify __defaults__ of the relevant methods yourself (assuming that the relevant parameters are exposed on the cq.Shape level).

[adam-urbanczyk]

AFAICT you could use gin.bind_parameter to do exactly what you need.

[bragostin]

That was indeed the solution, thank you @adam-urbanczyk ! For those who might have the same issue, here is the solution with gin_config: First I made a bash script to modify shapes.py to make SetRunParallel manageable by gin_config, launch cq-editor and revert back to the original shapes.py:

#!/bin/bash

FILE=$(find ~ -path /*/lib/*/site-packages/cadquery/occ_impl/* -name shapes.py)

FILES=$(echo $FILE | tr " " "\n")

for file in $FILES
do
   cp $file $file.bak
   sed -i '1 i import gin' $file # adds import gin at beginning of shapes.py
   sed -i '/def _bool_op/i\\ \ \ \ \@gin\.configurable' $file # adds @gin.configurable before _bool_op() function
   sed -i '/op: Union\[BRepAlgoAPI_BooleanOperation, BRepAlgoAPI_Splitter\]/a\ \ \ \ \ \ \ \ parallel=True,' $file # adds new parameter parallel=True to _bool_op() function
   sed -i 's/op.SetRunParallel(True)/op.SetRunParallel(parallel)/' $file # replaces default True value of SetRunParallel by gin_config managed variable parallel
done

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia cq-editor

for file in $FILES
do
   cp $file.bak $file
done

Then I created 2 gin_config files: optimizer.gin containing the line _bool_op.parallel=False and cadquery.gin _bool_op.parallel=True. Finally in my main script I added the 2 lines: gin.parse_config_file('optimizer.gin') where I need SetRunParallel to be False gin.parse_config_file('cadquery.gin') where I need SetRunParallel to be True

[adam-urbanczyk]

Glad that it helped. You could still open that PR adding the parallel parameter on the cq.Shape level, though.