mapnik / mapnik-gyp

GYP build system for Mapnik 3.x
8 stars 7 forks source link

AppVeyor: parallelize without running out of memory #26

Closed lightmare closed 8 years ago

lightmare commented 8 years ago

In short: run only 1 MSBUILD process, allow CL to use all processors, except on heavy sources where it's restricted to 1 (or 2 for heavier plugins)

It's possible to finish in 16 minutes: https://ci.appveyor.com/project/Mapbox/mapnik/history?branch=wip-msbuild-parallel

wilhelmberg commented 8 years ago

Great work, especially getting those heavy files more readable. I'll carry over some of your improvements, but I'm working on a version that requires less changes.

lightmare commented 8 years ago

The second commit (8f513b6 include common.gypi-appveyor conditionally) is completely optional, it's not needed for this to work.

As for the other two, the main criteria should be the level of parallelization, not the amount of changes. I don't run windows, so that's up to others to test, but here're some thoughts:

I added restrictive /MP1 or /MP2 flags on specific targets in mapnik.gyp (that's the first commit 2b8b57d detect AppVeyor inside mapnik.gyp, override /MP for hungry targets), and that allowed me to get rid of the second MSBUILD invocation. Now mapnik lib starts building immediately after the initial heavy sources compilation.