Faster plugins using multiprocessing

[SimonMerrett]

Hi, I am not a python-competent coder by any means but having said that, I would be prepared to contribute if the developers/maintainers of this fine repository would be able to support me doing so by helping get up to speed with the replication codebase and what the most fruitful routines to optimise would be.

I am watching a well specc'ed PC spend hours replicating a fairly large array of circuits in a matrix using only one of the 8 cores available to it. My understanding is that is due to the single thread limitations of Python and that multicore Python is possible for certain types of program using the multiprocessing module's Process and Pool classes.

Has this been considered in the past and would anyone be prepared to guide me on how best to make replication a multithread plugin? Link to an article which made me think it might be feasible.

---

EDIT: For example, I have just had a look at the replicator code and (bearing in mind that replicating footprints/modules and tracks is my main interest and seemed to take the longest durations on the progress bar) would it make sense to look at multiprocessing replicate_modules() and replicate_tracks() first?

[MitjaNemec]

With optimizations I prefer to follow the first rule of optimizations:

1. Don't do any optimizations

Now to get a bit more serious:

1. There is more to consider than python GIL. KiCad python interface is just internal C++ classes and methods exposed via SWIG. And I don't have a slightest idea what would happen if one would access KiCad internal objects concurrently. So this would need to be tested thoroughly as any threading/multi-processing issues are PITA to locate and bug show sporadically.
2. As multi-threading/multiprocessing is a big can of worms (to big in my opinion to start with), I'd first start with taking a look at the code and getting rid of all unnecessary iterations
3. Then I'd proceed with optimization by adding a bigger layout to test suite (standalone operation of replicatelayout.py via main function). Then I'd do some benchmarking via cprofile to look for the targets where the yield should be biggest.
4. Then and only then I'd consider multi-threading/multi-processing. But even then I'd really consider moving the plugin to main C++ code with cooperation of KiCad development team as tighter coupling should make this job easier.
5. Finally the timing of any code changes. I've stopped developing the plugin(s) for 5.1.x branch some time ago and I am currently only adding bug fixes, waiting what will happen with python API in 5.99/V6. When this is resolved I'll begin porting the plugin to 5.99/V6. So any improvements on 5.1.x code will need to be rebased to 5.99/V6 code. And this might be significant work. Furthermore I've got a nagging feeling that the functionality of the plugin in 5.99/V6 will not be the same as in 5.1.x(see issue 7982)

I hope I've presented my view, my reservations and general "lay of the land" well enough for you to understand. As such I'll not accept any PR in master branch (5.1.x) and any PR to 5.99_branch will be delayed until I port the plugin fully to stable 5.99/V6 python API.

[SimonMerrett]

@MitjaNemec thanks for that thorough appraisal. I am happy to hold back and grateful for the insight. I may be better off making a python standalone script to parse an extracted set of traces/modules into a separate pcbnew file in the array desired.

[MitjaNemec]

I may be better off making a python standalone script to parse an extracted set of traces/modules into a separate pcbnew file in the array desired.

I did not get this. Could you elaborate a bit on your workflow. I think that your workflow is outside of the main plugin intention

[SimonMerrett]

Sure - I meant to completely ignore all plugin workflow, take a pcbnew file containing a single instance of each track and module I want and then use python to write a new kicad pcb file containing multiple instances of each module and trace, with the coordinates adjusted. Thank the Lord for ascii file formats!

[MitjaNemec]

I don't see how this is connected to the speed of my plugin.

From my experience, I've found out that I'd appreciate faster execution only when the layout requires a lot of iterations (lay out one sheet, replicate, modify one sheet, replicate, modify one sheet, replicate, ....) where layout of source sheet depends on the layout of destination sheets. In these cases you often need to also tick delete duplicates checkbox, which is really a resource hog.

[SimonMerrett]

My plan going forward is not to do with your plugin - I only mentioned what I am planning to do out of interest in case someone else wants to do something similar to me and can't think of any alternative. Sorry for the confusion.

I have over 500 copies of a particular sheet to lay out. I am manually deleting tracks and zones with the selection filter before running replicator so as to avoid using the delete duplicates option.

[MitjaNemec]

No worries. Part of communicatio.

With 500 copies I can see why the need for speed.

I've never had a plugin user eport more than 10 copies.