search

heitzmann / gdstk

Gdstk (GDSII Tool Kit) is a C++/Python library for creation and manipulation of GDSII and OASIS files.
https://heitzmann.github.io/gdstk/
Boost Software License 1.0
319 stars 79 forks source link

Multi-threading #8

Open heitzmann opened 3 years ago

heitzmann commented 3 years ago

It should be possible to increase the performance of Cell.to_gds using multi-threading. The generation of polygonal boundaries from FlexPath and RobustPath and fracturing all polygons (including the results from paths) could all be done in parallel. Furthermore, the left and right sides of the polygonal boundaries from paths are independent and could also be created in parallel.

There is a possible problem with increased memory use if multiple paths are created in parallel. In this case, we could give the user an option to run single-threaded or, even better, try to auto-detect the issue and limit the number of threads.

tvt173 commented 2 years ago

curious what a typical breakdown is on the time for computational tasks at write time you list here like fracturing, etc. vs IO. do you know? if they are same order of magnitude, you may be able to get a good first cut at this by making just the IO part asynchronous. it will also tell you your potential gain if you multithread the computational part.

heitzmann commented 2 years ago

I believe writing to the disk is the slowest part by far, typically. Fracturing operations are not that expensive in general, and path conversions (when needed) can be slow for complex paths.

The tricky part is defining “typical”. Our research group works mostly on optical device design, so our layouts are usually very small, albeit with irregular shapes. My guess is a typical photonics layout would have lots of curved paths/shapes that should require fracturing, but a microelectronics might be mostly rectangles and simple paths. In those cases, IO is probably the bottleneck. But there are many exceptions, for example, a lage photonic crystal area or the inversion of a large layout would have incredibly large and complex polygons, in which fracturing would start showing up in a time breakdown.

The main problem is that if we're limited by the hardware write speed, issuing parallel writes to the same hardware will decrease performance, instead of increasing.

rocallahan commented 3 days ago

issuing parallel writes to the same hardware will decrease performance, instead of increasing.

With SSDs this is not necessarily true; it is possible to write in parallel very efficiently. However it's not trivial, e.g. you might have to use direct I/O and block-sized writes.