search

KLayout / klayout

KLayout Main Sources
http://www.klayout.org
GNU General Public License v3.0
754 stars 196 forks source link

strmxor needs 40GB of memory to xor a 2500x2500 hole fence #1636

Open stefanottili opened 4 months ago

stefanottili commented 4 months ago

While testing worst case scenario's for scanlines I noticed that strmxor uses a lot of memory when encountering "fence" data. Input data which causes results with lot's of holes have been seen in the wild as either cad layers in ram/rom generators or as shielding for analog blocks. This is just a "rare corner case" performance issue report, given enough memory strmxor does the right thing.

The attached fence with 2500x2500 holes requires up to 40GB of memory, even when xor'ing itself, not generating any output. One horizontal, one vertical array with 2500 row/col, referencing a path.

strmxor --debug-level=21 --threads=8 --heal --tiles=500 test_xor_fence.oas test_xor_fence.oas xor.oas

strmxor ... --layer-details test_xor_fence.oas empty.oas xor.empty.oas creates a single polygon with a large number of edges.

xor.empty.gds breaks the polygon into smaller pieces, but klayout throws Warning: Record length larger than 0x8000 encountered: interpreting as unsigned (position=118, record number=9, cell=XOR) since the --max-vertex-count default is 8190, a smaller value get's rid of this warning.

It would be nice if there would be an oas equivalent to the gds specific --max-vertex-count test_xor_fence.oas.gz empty.oas.gz

stefanottili commented 4 months ago

One strategy to avoid excessive memory consumption is to monitor the scanline memory requirements. If they exceed a certain threshold, stop operation and divide the chip/tile recursively into smaller pieces til they're small enough to run. And in case the output polygons have "too many" edges, switch off healing.

klayoutmatthias commented 3 months ago

I will try to find a solution, but a need a little time. It will not make it into 0.29.

stefanottili commented 3 months ago

Given enough memory, klayout does the right thing, the problematic case is so unusual that just skipping xor'ing this particular layer is good enough, see https://github.com/KLayout/klayout/issues/1668

But in order for that to be useful, strmxor and klayout would have to change their "foreach tile, foreach layer" processing to "foreach layer, foreach tile", so that a user get's feedback which layer is causing xor to take a lot of time/memory.