Implement PETsc Python modules for the physics solves

[dirkmunro89]

I run a topopt problem, and I do an HTOP, and I see only one core working very hard.... Is the Python Petsc modules not a nice way to go for the matrix solves...?

[dirkmunro89]

https://computationalmechanics.in/vector-and-matrix-operations-in-petsc-via-petsc4py/

[aatmdelissen]

The petc4py is pretty hard to get installed and working correctly. Also, all (sub)solvers will have to be made such that they work correctly in parallel, which is a tough job! Nexto to that, the focus of this repository is not to do large-scale topology optimization, but to provide a playground/sandbox to experiment with different optimization settings. Users may use our optimization schemes for their own large-scale code though.

Your comment being noted, it may be a good idea to switch solvers. I think we are using the scipy sparse solver now, which is very slow; about a factor 10 compared to Matlab's backslash \. We might be able to include the CHOLMOD solver (which is used for Matlab as well): https://pypi.org/project/scikit-sparse/.

[dirkmunro89]

OK; just to be clear: I mean petsc NOT for the subsolves; I mean, PETSC for parallelization of the physics; right?

[aatmdelissen]

Yes, if your physics discretization is in parallel, your design variables are also distributed in parallel. You would have to build in a gather operation to the main proc, do the optimizer subsolve, and subsequently do a scatter. This will probably just shift the bottleneck from FEA solutions to the subproblem solutions.

[dirkmunro89]

Yes indeed; we are on the same page. Not so sure about the last sentence though; pending on the problem, it may balance in advantages ways (see the silly AM parallelization work I did right next to you at the TU! :) ). There I gathered and distributed in the 'simulation' routine (upon entering and before existing); the rest of the algorithm was not parallelized. Also, I solved a problem with one constraint, and I exploited this with a dual QP formulation.

[aatmdelissen]

Yeah I agree that up to some point there will be a benefit in parallelizing the physics simulation. However, the computational benefit does (at this point) not outweigh the effort to implement. PETSc(4py) uses its own data structures and matrices, so we'd have to implement the filtering scheme, FE assembly, etc. once more, but then in parallel with PETSc matrices and vectors.

[dirkmunro89]

Yes; it could be a change to filtering, as well.... but not necessarily... One can imagine for example the equivalence with calling an external analysis (like Abaqus). There, also, there is a complete 'translation' into a completely different data structure (and retranslation of it into the algorithms data structure)... and filtering can be applied, as a first step, to the algorithm native data structure... Its not super important / I am just thinking out loud. I might try it / see what is the simplest way to do it, and whether that makes sense, just for fun.

[dirkmunro89]

*but likely for large scale, at least the filtering would need parallelization as well; else it just becomes the bottle neck.