search

ThisIsNotRocketScience / GerberTools

Tools to load/edit/create/panelizer sets of gerber files
MIT License
725 stars 144 forks source link

Speed improvements #131

Closed chrissnow closed 3 years ago

chrissnow commented 3 years ago

We have been using your excellent gerber panelizer for a while now and it's been great, it is however rather slow on bigger boards some are now 4 hours plus.

I'm happy to invest some time to make it better but may need some help\pointers. First observation is that the processing seems to be single threaded? could a simple thing improvement be to process each layer in parallel threads? each layer should not have any impact on another so should be thread safe in theory?

Any help would be appreciated, but happy to give it a go myself and PR when suitable.

StijnKuipers commented 3 years ago

4 hours! wow.. Could you give me some stats on the type of panels you are processing? How many subboards, boardarea etc?

Some stuff may indeed be as simple as a parallel.for in strategic places - for our Fenix4 setup I did exactly that. Not so much "per layer" as "per board" first though - rotation/translation is done on all boards first - then afterwards the layers are gathered and merged - this can indeed also be parallel.

PRs always welcome ofcourse - would love to see some of these monster panels!

On Tue, 9 Feb 2021 at 16:22, Chris Snow notifications@github.com wrote:

We have been using your excellent gerber panelizer for a while now and it's been great, it is however rather slow on bigger boards some are now 4 hours plus.

I'm happy to invest some time to make it better but may need some help\pointers. First observation is that the processing seems to be single threaded? could a simple thing improvement be to process each layer in parallel threads? each layer should not have any impact on another so should be thread safe in theory?

Any help would be appreciated, but happy to give it a go myself and PR when suitable.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ThisIsNotRocketScience/GerberTools/issues/131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABSI6MJQEE5PE3CTZENMOLS6FHMXANCNFSM4XLGBAJQ .

harry-bridge commented 3 years ago

Here are the gerbers we are trying to panellise, nothing too exciting unfortunatly. Have a look for yourself all the files should be there. It's been running now for ~7 hours, its running in a windows VM with 6gb ram and 4 cores on an i9 processor.

LORA Corecell V3.zip

StijnKuipers commented 3 years ago

Did some debugging - some weird memorythrashing was happening, full panel export time (single threaded) is now back to minutes instead of hours

I'll push the fix when Ive checked nothing broke with this. Are you OK building from source or do you want a new binary release?

On Tue, 9 Feb 2021 at 17:07, Harry Bridge notifications@github.com wrote:

Here are the gerbers we are trying to panellise, nothing too exciting unfortunatly. Have a look for yourself all the files should be there. It's been running now for ~7 hours, its running in a windows VM with 6gb ram and 4 cores on an i9 processor.

LORA Corecell V3.zip https://github.com/ThisIsNotRocketScience/GerberTools/files/5952532/LORA.Corecell.V3.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ThisIsNotRocketScience/GerberTools/issues/131#issuecomment-776052227, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABSI6LT5MAA2XVLLZ33MBLS6FMSVANCNFSM4XLGBAJQ .

harry-bridge commented 3 years ago

That's brilliant thank you for your effort and quick fix! Building it from the source is fine as that's what we've been doing thusfar as you've been doing updates

StijnKuipers commented 3 years ago

Pushed!

Thanks for the bugreport! This will no doubt make life easier for a lot of people :-)

On Tue, 9 Feb 2021 at 17:36, Harry Bridge notifications@github.com wrote:

That's brilliant thank you for your effort and quick fix! Building it from the source is fine as that's what we've been doing thusfar as you've been doing updates

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ThisIsNotRocketScience/GerberTools/issues/131#issuecomment-776070329, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABSI6MDWNISMSYDZRSOV7LS6FQBZANCNFSM4XLGBAJQ .

chrissnow commented 3 years ago

Apologies for the delayed feedback...

Your changes worked perfectly and we are now back to a much more bearable few mins per run :-)

Thanks your help to resolve it.